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Executive Summary 


Introduction 


Throughout the 2018-19 school year, 18 middle school teachers and five 
administrators from three U.S. school districts partnered with instructional 
coaches and learning sciences researchers from Digital Promise to address 

an ambitious educational challenge: How might we deepen engagement and 
learning of middle school science in our schools and beyond? The overarching 
goals of the Challenge Based Science Learning Project, funded by the William 
and Flora Hewlett Foundation, included producing high-quality open educa- 
tional resources (OER) for middle school science, and understanding whether 
and how activities built around these resources can promote deeper learning 
in science classrooms. Digital Promise organized this collaborative effort and 
also provided formative evaluation and research support. 


The project's main activity was for teachers to create and implement science 
lessons that were “challenge based" and aligned with the Next Generation 
Science Standards (NGSS) (NGSS Lead States, 2013). Challenge Based Learning 
(CBL) is a type of problem-based learning distinguished by its requirement that 
students engage with, investigate, and act on authentic challenges they find 
personally meaningful (Cator & Nichols, 2008; Nichols, Cator & Torres, 2016). 
In accordance with this model, participating teachers designed science lessons 
where teachers prompted students with a phenomenon and had them come 
up with questions and a challenge to pursue. To align with NGSS, teachers 
identified in advance the set of disciplinary core ideas, science and engineering 
practices, and cross-cutting concepts that the challenges would target. 
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The project's overall goals were to promote deeper learning in middle school 
science lessons and to strengthen the ecosystem of K-12 OER by drawing from 
and contributing to the existing pool of resources and supports. The project 
was experimental in nature: No one to our knowledge had tried combining 
CBL and NGSS in this way, or examined the impact of this combination on stu- 
dent learning. Furthermore, it was unclear whether a curriculum design project 
could impact use and development of OER in K-12 contexts. Synthesizing what 
we learned from 54 teacher interviews, more than 150 hours of classroom 
observations, 32 student focus groups, 71 samples of science lessons and as- 
sociated student work, and participatory observations and participant surveys 
of two multi-day lesson design workshops, this report addresses the 

following questions: 


1. What do teacher-created challenge based lessons aligned to 
NGSS look like? 


2. If teachers make and implement challenge based lessons that are 
aligned to NGSS, does deeper learning occur more frequently? 


3. What are lessons for the field? What are facilitators and barriers for 
standards-aligned challenge based learning? To what extent and in 
what ways do OER facilitate standards-aligned, challenge based 
science learning? 


Characteristics of the Challenge Based Science Learning Lessons 


Both CBL and NGSS focus on student-driven problem solving, with NGSS 

emphasizing three-dimensional learning of science, and CBL bringing student 
self-direction and engagement to the fore. Combining 

the pedagogical ideals of CBL and NGSS, teachers in this Both CBL and NGSS focus on 


project produced 15 challenge based science lessons Ele -laierelgivcae dgelel leak late) 


that had several features in common. They centered with NGSS emphasizing three- 


around complex, real-world issues; they involved rounds dimensional learning of science, and 


of student questioning and student-led research (usually CBL bringing student self-direction 


online, secondary research); and they involved students 


and engagement to the fore. 


learning and working toward a goal greater than simply 
“learning the material” or “getting a grade." 


The lessons ranged from examining human impact on the environment to 
engineering solutions and addressing public health concerns. In each lesson, 
students typically engaged with an essential question about helping others 
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through raising awareness or designing innovations. Teachers often encour- 
aged students to generate questions about an intriguing phenomena to help 
them internalize the lessons’ essential questions, and take ownership of their 
challenge. Students’ investigations of their essential question fell under three 
main categories. All lessons involved some amount of secondary research, 
where students searched existing information to better understand phenome- 
na and/or inform their design solutions. Many students also conducted science 
investigations and used engineering design practices to solve problems. Finally, 
the lessons called for students to take action and rise to their challenge, with 
most student action steps having to do with awareness raising, or with con- 
ceptualizing products to help the environment or improve the 

human condition. 


When possible, teachers used open access resources to ensure that the les- 
sons they created could be shared with other teachers without any copyright 
barriers. While teachers co-created a resource bank of existing middle school 
science OER, and were provided some time to explore these resources, most 
lessons teachers created did not rely substantially on existing OER in either de- 
sign or conception. Teachers relied most heavily on their personal knowledge 
and experience with their students as well as practical considerations (e.g., 
taking into account the content that needs to be covered when the lesson will 
be implemented) to decide on their lesson topic. Once the lesson topic was 
decided, teachers typically used Google search to find relevant materials rather 
than looking through different OER websites. 


Many aspects of the challenge based science lessons felt new and beneficial to 
most teachers, particularly student ownership of learning and the engagement 
that comes with it. Most students also saw the challenge based lessons as 
being very different from their normal lessons and appreciated the lessons’ 
real-world relevance, the freedom to learn, and working with others. Teachers 
identified several areas in which the lessons could be improved, including 
sequencing and logistics, assessment, and promoting deeper research skills. 
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Deeper Learning Opportunities and Outcomes in 
Challenge Based Lessons 


The project compared challenge based lessons with lessons that teachers 
characterized as representative of what students usually do in their classes 
(“typical”), and lessons that were relatively well-aligned with at least one of 
the NGSS science and engineering practices ("NGSS practices-aligned’). 
Independent scorers scored the lessons and associated student work on ru- 
brics that were designed to assess deeper learning and grade-appropriateness 
of science content. 


The results were highly encouraging with respect to the potential for chal- 
lenge based science lessons to provide deeper learning opportunities to 
students. On average, relative to typical lessons, challenge based science 
lessons substantially provided more opportunities for: real-world engagement, 
self-direction, practice of science and engineering skills, conducting critical 
research, substantive collaboration, and effective communication. Relative to 
NGSS practices-aligned lessons, the challenge based lesson scores were rated 
significantly higher in three of the six deeper learning dimensions (real-world 
engagement, critical secondary research, and effective communication) and 
equivalent in the remaining three areas (self-direction, practicing science and 
engineering, and substantive collaboration). Where challenge based lessons 
scored higher, they score higher by quite a lot (between 0.81 to 1.75 standard 
deviation units). Moreover, challenge based lessons did this while being just as 
strong as the two other types of lessons in terms of providing opportunity to 
learn grade-appropriate science content. Student work scores showed, unsur- 
prisingly, that students’ deeper learning outcomes are positively and strongly 
correlated with rating of the opportunities to learn deeply. It also confirmed 
teachers’ and students’ reports that students engaged more with the real 
world, did more secondary research, and communicated more often and more 
effectively in challenge based lessons. 


These results also indicated that there is room for improvement in the chal- 
lenge based lessons, especially in two areas: increasing opportunities for 
students to practice science investigations and engineering design, and to 
conduct secondary research more critically. Corroborating what the research 
team observed through classroom visits, the independent scorers who exam- 
ined the challenge based lessons found that the deeper learning opportunities 
in these two areas, while stronger than the opportunities in typical lessons, 
were still just “emerging” (i.e., a score of approximately 1). So while challenge 
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based lessons designed in this project significantly provided “more” opportuni- 
ties than typical lessons (that scored on average approximately or a little over 
0), the challenge based science lessons left room for improvement on 

this front. 


Lessons for the Field 


Many of the project teachers said their key takeaway was that it was difficult 
but worthwhile to let students drive some of their own learning students 

can rise to learning challenges when given the opportunity, and this type of 
learning can benefit all students, not just high achievers. Teachers found the 
lesson design process to be difficult because they had to step out of their 
comfort zone in many ways, and so much was left for them to figure out. They 
recommended that other teachers could benefit from having some examples 
of strong NGSS-aligned CBL units to start from. Finally, two teachers who were 
fully trained in project-based learning speculated that incorporating short- 
er-term CBL units, as teachers did in this project, might be a more scalable 
alternative than trying to shift teachers to doing all of their instruction 

through projects. 


Students had advice as well, mainly for their teachers. They wanted the proj- 
ects to be relevant and wanted enough time to work on them. They expressed 
a preference for doing “hands-on” activities, such as experiments and proto- 
type building, rather than just reading about things on the internet. 


We noticed, in addition, that student-centered, standards-aligned science 
teaching requires many skills, underscoring the importance of creating a rich 
learning ecosystem for teachers where these skills can be cultivated and de- 
veloped. Particularly when a large transformation of values, beliefs, or behav- 
iors is requested of teachers, professional learning experiences should: build 
on teachers’ knowledge and expertise; be experiential, intense, and affective; 
be professionally relevant; be organized around value-laden goals; provide 
sustained opportunities for critical reflection; and be founded on 

authentic relationships. 


Next Steps 


The NGSS-aligned CBL units created through this project provide the content 
for a repository of middle school science materials Digital Promise has made 
available under a Creative Commons license. Future professional learning 
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activities around student-centered, NGSS-aligned instruction can make use of 
these examples. A refined version of the rubrics used to rate deeper learning 
opportunities provided by the units and displayed in student work was also 
created. (Several of the rubric level definitions were refined based on sugges- 
tions from raters as well as reliability indices.) 


Published examples of CBL science units and deeper learning rubrics can raise 
interest and awareness. However, to really achieve high-quality implementa- 
tion of challenge based science learning, an investment in professional learn- 
ing and support systems for science teachers is necessary. Despite national 
efforts to promote NGSS, less than half of all middle school science teachers 
emphasize learning how to do science in their classrooms and less than 10 
percent emphasize learning how to do engineering. Sporadic half-day profes- 
sional development sessions are not going to be sufficient to overcome the 
present over-reliance on teacher-directed transmission of science concepts. 
While not all teacher professional development can (or necessarily should) 

be as extended and participant centered as the teacher learning experiences 
in this project, initiatives that encourage teachers to radically change their 
instruction to a more student-centered approach will always need a significant 
experiential component. 


The next logical line of inquiry involves figuring out a cost-effective approach 
to supporting teacher learning and culture change to emphasize more active, 
Challenge Based Learning. Prior experience would suggest that giving stu- 
dents encouragement and access to existing, NGSS-aligned CBL units would 
be insufficient to support high-quality implementation by most teachers. 

But it may not be necessary for teachers to design their own CBL units from 
scratch as they did in this project. A hybrid approach, in which teachers have 
examples of well-designed units and unit templates but engage in customizing 
the unit templates for their own students and curriculum, could reduce time 
requirements and teacher effort while still engendering a sense of ownership. 
We believe it is important, though, that any streamlined version of professional 
development around challenge based science learning preserve Key qualities 
of this project, including leadership support, modeling of the target instruc- 
tional approach, an ongoing learning community, and multiple iterations with 
feedback and reflection. 
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Section 1. About the Challenge Based Science 
Learning Project 


Throughout the 2018-19 school year, 18 middle school teachers and five administrators 
from three US school districts, serving diverse student populations,’ partnered with instruc- 
tional coaches and learning sciences researchers from Digital Promise to address an am- 
bitious educational challenge: How might we deepen engagement and learning of middle 
school science in our schools and beyond? The overarching goals of the Challenge Based 
Science Learning Project, funded by the William and Flora Hewlett Foundation, included 
producing high-quality open educational resources (OER) for middle school science, 

and understanding whether and how activities built around these resources can promote 
deeper learning in science classrooms. Digital Promise organized this collaborative effort 
and also provided formative evaluation and research support. 


As described in Table 1, the project incorporated a number of important big ideas in 
education, including Challenge Based Learning (CBL), deeper learning, three-dimensional 
science learning and open educational resources. The project's main activity was for 
teachers to create and implement science lessons that were “challenge based” and aligned 
with the Next Generation Science Standards (NGSS) (NGSS Lead States, 2013). CBL is a 
type of problem-based learning distinguished by its requirement that students engage 
with, investigate, and act on authentic challenges they find personally meaningful (Cator & 
Nichols, 2008; Nichols, Cator & Torres, 2016). In accordance with this model, participating 
teachers designed science lessons where teachers prompted students with a phenomenon 
and had them come up with questions and a challenge to pursue. To align with NGSS, 
teachers identified in advance the set of disciplinary core ideas, science and engineering 
practices, and cross-cutting concepts that the challenges would target. 


Most teachers worked in teams of two or three, co-designing the lesson with others 
who taught the same grade at the same school. To support teachers’ design efforts, the 


+ Teachers taught in five socioeconomically diverse public middle schools in three districts, serving grades six through eight. Eight teachers 
taught in Title | schools, with one school having high proportions of White and Black students, and the other predominantly Southeast Asian 
and Hispanic. Others taught in relatively affluent schools where the student population was either predominantly White, predominantly 
Southeast Asian, or predominantly White and South Asian. Enrollment was approximately 450 per grade, except in one school with 
approximately 250 per grade. Districts were recruited through a national network of forward-thinking superintendents and selected based on 
their stated interest and capacity to develop and implement challenge based NGSS-aligned curricula. 
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project's initial two-day workshop introduced NGSS and CBL concepts, and provided 
models of CBL instruction and resources for the teacher teams to use in designing their 
lessons. The second workshop provided opportunities for teachers to reflect together on 
their first experience with such lessons and to design a second lesson. District leads not 
only participated in the workshops and helped provide professional development time and 
resources, but also supported the teachers instructionally and emotionally (Jordan, 2019). 


The research team conducted teacher interviews, classroom observations, and student 
focus groups both before and after teachers attended the design workshops. The project 
timeline is provided in Table 2. 


Table 1. Big Ideas That Informed the Challenge Based Science Learning Project 


Big Idea 


Challenge 
Based 
Learning 
(CBL) 


Briefly, what it is 


A type of problem-based learning 
where students learn by identifying 
and rising up to an authentic challenge 
in their community. CBL requires 


students to engage with, investigate, 
and act upon their chosen real-world 
challenges. See CBL website and CBL 


Publications and Research for more. 


“[Aln umbrella term for the skills 

and knowledge that students must 
possess to succeed in 21st century 
jobs and civic life. At its heart is a set of 
competencies students must master in 
order to develop a keen understanding 
of academic content and apply 

their knowledge to problems in the 
classroom and on the job” 

(Hewlett, 2013). 


Deeper 
Learning 


Three- 
Dimensional 
Science 
Learning 


“Three-dimensional learning shifts 

the focus of the science classrooms 

to environments where students use 
disciplinary core ideas, cross-cutting 
concepts with scientific practices to 
explore, examine, and explain how and 
why phenomena occur and to design 
solutions to problems" (Krajcik, n.d.). 


Open 
Educational 
Resources 


“Teaching, learning and research 
materials in any medium-—digital or 
otherwise-that reside in the public 
domain or have been released 
under an open license that permits 
no-cost access, use, adaptation and 
redistribution by others with no or 
limited restrictions” (Hewlett, n.d.). 


Why it was important for the 
project 


Middle school is an important 

period when students become more 
self-conscious, socially aware, and 
independent-minded. We wanted to 
support students in their developmental 
growth trajectory by having them start 
to drive their own learning and connect 
what they were learning with the world 
around them. CBL provided a specific 
pedagogy to do this. 


We wanted students to learn science 
more “deeply” rather than just memorize 
facts and skills. The deeper learning 
scholarship helped identify what we 
might want to attend to in student 
learning, beyond test scores. 


We wanted students to learn science in 
ways that are supported by the leading 
science education standards and that 
would support their success with any 
assessments that might be aligned to 
these standards. All districts participating 
in the project were located in states that 
had adopted or were soon to adopt 

the NGSS. 


We wanted any lessons and ideas that 
were created from this project to be 
easily and freely shared with other 
teachers around the world. 


Digital Promise 
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Table 2. Challenge Based Science Learning Project Timeline 


Baseline Design First Design Second 
Visits Workshop #1 | Implementation |Workshop #2 | Implementation 
Round Round 


Nov 2018 Dec 2018 Jan-Feb 2019 Mar 2019 Mar-May 2019 


Design Baseline visit Introduction to | Design first Explore OER Implement 
Activities | to introduce CBL and NGSS_| challenge based and deeper second challenge 
the project and science lesson learning based science 


learn about Design first lesson 
the learning challenge Design second 


context based science challenge 
lesson based science 
lesson. 


Research | Teacher Participatory Observations, Participatory Observations, 

Activities | interviews, observation, teacher interviews, | observation, teacher interviews, 
observations, perception student focus perception student focus 
collection of surveys groups, collection | surveys groups, collection 
lessons and of lessons and of lessons and 
student work student work student work 


As noted above, the project's overall goals were to promote deeper learning in middle 
school science lessons and to strengthen the ecosystem of K-12 open educational re- 
sources (Huttner, Green, & Cower, 2018) by drawing from and contributing to the existing 


pool of resources and supports. The project was experimental in nature: No one to our 
knowledge had tried combining CBL and NGSS in this way, or examined the impact of this 
combination on student learning. Furthermore, it was unclear whether a curriculum design 
project like this could impact use and development of OER in K-12 contexts. The research 
aspects of the project addressed the following questions: 


1. What do teacher-created challenge based lessons aligned to NGSS look like? 


2. If teachers make and implement challenge based lessons that are aligned to NGSS, 
does deeper learning occur more frequently? 


3. What are lessons for the field? What are facilitators and barriers for standards-aligned 
CBL? To what extent and in what ways do OER facilitate standards-aligned, chal- 
lenge based science learning? 


Our report synthesizes what we learned from 54 teacher interviews, more than 150 hours 
of classroom observations, 32 student focus groups, and participatory observations and 
participant surveys of two multi-day lesson design workshops. Furthermore, to better 
understand deeper learning opportunities and outcomes, we collected and analyzed 71 
samples of middle school science lessons and associated student work—both challenge 
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based lessons and not. The lessons were scored by independent scorers, using rubrics that 
the research team developed for the project. One set of rubrics concerned deeper learning 
opportunities that the lesson or activity provided to students and consisted of rubrics for 
six dimensions of deeper learning opportunities. Another set was for rating evidence of 
deeper learning that could be seen within the student work and consisted of rubrics for five 
dimensions. More details about the scoring process and results are provided later in this 


report (Section 3, Appendix A) as well as in a technical report by lwatani, Vang, Romero, and 
Means (forthcoming). 
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Section 2. What Did the Challenge Based Science 
Learning Lessons Look Like? 


Common features of challenge based science lessons and their 
connections to NGSS and CBL 


Consensus around critical features of NGSS-aligned lessons has emerged (Next Generation 


Science Standards, 2016). First, the lessons should focus on students using science and 


engineering practices to explain phenomena or design solutions to a problem. In addition, 
lessons should focus on helping students develop all three dimensions of the standards— 
science and engineering practices, disciplinary core ideas, and cross-cutting concepts—at 
the appropriate grade level and in an integrated way. Ideally then, students are learning 
science content, skills, and cross-cutting ideas all at the same time as they explain science 
phenomena and engineer solutions. High-quality science lessons, according to the NGSS, 
must also be authentic and relevant, provide ample opportunities for students to commu- 
nicate and refine their ideas, and explicitly identify and build on students’ prior learning in 
all three dimensions. 


Some of these core features of NGSS-aligned lessons overlap with central features of 
high-quality challenge based learning lessons. Challenge based lessons must be relevant, 
driven by student curiosity, involve substantial student choice, foster student self-direction, 
and empower students to act to help others or provide value to their community. Thus, 
both frameworks focus on student-driven problem solving, with NGSS emphasizing 
three-dimensional learning of science, and CBL bringing student self-direction and en- 
gagement to the fore. 


Combining the pedagogical ideals of CBL and NGSS, teachers in this project produced 

15 challenge based science lessons that had several features in common (Table 3). They 
centered around complex, real-world issues; they involved rounds of student questioning 
and student-led research (usually online, secondary research); and they involved students 
learning and working towards a goal greater than simply “learning the material” or 
“getting a grade." 
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Table 3. Common Lesson Features of Challenge Based Science Lessons 


Centers Around Complex 
Real-World Issue: Students are 
introduced to a complex, real- 
world issue that defies simple 
explanations and solutions. 


Questioning: Students arrive 
at a question of personal 
importance, after extended 
questioning about a real-world 
problem or phenomenon of 
personal relevance. 


Self-Guided Research: Students 
guide their own learning to 
better understand and act 

upon their essential question. 
This invariably involves some 
online research, and sometimes 
involves science investigations 
and engineering design 
processes. 


Learning and Working Toward a 
Greater Goal: Students address 
their essential question. 


Relevance, curiosity 


Relevance, curiosity, student 
choice, student self- 
direction, acting to 

help others 


Relevance, curiosity, 
student self-direction 


Relevance, student self- 
direction, acting to 
help others 


What the lessons looked like in practice 


Precursor to explaining 
phenomena or designing 
solutions, anchoring 
phenomena 


Anchoring phenomena, asking 
questions (a science and 
engineering practice) 


Three science and engineering 
practices: obtaining and 
communicating information, 
planning and carrying out 
investigations (in some lessons), 
and constructing explanations 
and designing solutions 


Designing solutions 


Teachers spent a great deal of care and time in selecting their lesson topics, with many 


reporting difficulty identifying a topic that both engaged students and incorporated all the 


content and skills that needed to be addressed. The resulting lessons ranged from exam- 


ining human impact on the environment to engineering solutions and addressing public 


health concerns. In each lesson, students typically engaged with an essential question 


about helping others through raising awareness or designing innovations (Figure 1). 


Digital Promise 
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Figure 1. Challenge Based Science Lesson Topics 


Addressing human impact on the environment ————— 


Global Protecting our 
warming native species 


Pollinators and 
our food 


California fires 


Water pollution 


Finding a dream home for trout 


Engineering solutions 


Building a cell At 
phone jail A 0'g) 


@ O Earthquakes—prevent, 
e 


Structures that ? 
How blimps stay afloat ae five Bs e O predict, prepare 
the moon O e Creating something useful 
Special Olympics and that subtracts thermal energy 


assistive technology 


Addressing ty 


Adverse impacts Amitai 2 i aa ea 
of blue light resistance public health — Building a Stonehenge trilithon with 
cOnee tS simple machines 


Figures 2 and 3 illustrate two challenge based science lessons created through this project. 
More examples of lessons are provided in “CBL for NGSS toolkit” (2020). In keeping with 
the CBL framework, in each lesson students engaged with an essential question and 
challenge, conducted investigations, and took some action to make a difference. 
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Figure 2. Example #1 of Challenge Based Science Lessons Created 


Example Lesson Flow 


Engage Learn that pollinators are really really 
important to food 
Challenge: Educate public or create 
solution on importance of pollinators 


Investigate Science investigations 


¢ Pollinator simulation activity (see why 
pollinators die) 


e Flower dissection (see 
parts/mechanisms of pollination) 


Secondary research 


¢ Individual research on their pollinators 


Act Pollinator convention 


Figure 3. Example #2 of Challenge Based Science Lessons Created 


Example Lesson Flow #2 


Engage Moon-landing anniversary & NASA 
wants engineers to design structures 
that will allow humans to live on 
the moon 
Challenge: Design a structure that'll 
help us live on the moon 


Investigate Define 


e Empathize with the issue (what do we 
need to know?) 


e Secondary research on the questions 


¢ Science investigation: 
Moon-sun-earth simulation 


Develop solutions 


¢ Work in groups to come up with blue 
print & explanation 


Act Create/submit blue print & explanation 


Engaging with the essential question 


Teachers used different strategies to help students internalize and take ownership of 

the lessons’ essential questions so that they would be motivated to engage with their 
challenge. Many teachers decided to do this by introducing students to an intriguing 
phenomenon related to the lesson topic (e.g., through storytelling, showing a video or 
images, reading a news article, or inviting a guest speaker) and letting students’ questions 
about the phenomenon drive the rest of the lesson. This is an approach commonly used in 
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conjunction with inquiry approaches to science education (e.g., Next Generation Science 


Storylines, CREATE for STEM, inquiryHub). Students were often asked to write down their 
questions on sticky notes. These questions were then pooled, shared, and categorized 
either as a whole class or in small groups. In some lessons, teachers guided students to 
come up with the essential question through a questioning process, while in other lessons 
the teachers used the questioning process to get students to begin to respond to the 
essential question. 


Three kinds of student investigations 


Students’ investigations of their essential question fell under three main categories. All 
lessons involved some amount of secondary research, where students searched existing 
information to better understand phenomena and/or inform their design solutions. For 
example, eighth graders who were trying to help their schoolmates more comfortably 
participate in the Special Olympics looked for information on a variety of topics, such as 
“What is the Special Olympics?” and “What are the causes/symptoms of the special needs 
conditions that our friends have?” 


Students also conducted science investigations, or activities that allowed them to discover 
scientific principles through firsthand experience in contrast to reading and summarizing 
what has already been discovered. These included dissections, simulations, experiments, 
and field work. For example, seventh graders who were educating others about the impor- 
tance of pollinators dissected flowers to deeply understand what pollen looks and feels like 
and how it is related to the rest of the flower. Students also participated in a bee simulation 
that allowed them to “see” how environmental and human factors led to increases and 
decreases in the bee population. Not all science lessons involved science investigations 
(i.e., some lessons involved only secondary research), however, which meant that students 
lacked opportunities to engage in the NGSS science and engineering practices involved in 
conducting primary investigations (e.g., planning and carrying out investigations, analyzing 
and interpreting data, engaging in argument from evidence, and using mathematics and 
computational thinking). This is an area that could and should be improved in future 
iterations of this type of professional learning initiative. 


The third investigation type was students’ use of engineering design practices to solve 
problems. NGSS construes “engineering” broadly, “to mean any engagement in a sys- 
tematic practice of design to achieve solutions to particular human problems” (National 
Research Council, 2012, p.11-12). NGSS also maintains that the general practices of 
engineering design (“defining and delimiting engineering problems,” “designing solutions to 
engineering problems,” and “optimizing the design solution”) should be taught broadly in 


science classes because they are “practices that all citizens should learn” (Next Generation 
Science Standards, 2013, p.1). Engineering design shares commonalities but “has a different 


purpose and product than scientific inquiry” (Next Generation Science Standards, 2013, 


p.1), namely that engineering is about using a design process to develop solutions, while 
the goal of science inquiry is to understand and explain phenomena. Learning one kind of 
practice does not necessarily imply learning of the other, although scientific inquiry is often 
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an essential aspect of engineering (e.g., conducting experiments to see what materials 
work best for a thermos), and engineering of new tools or processes is often essential to 
new science discoveries (e.g., the importance of X-ray crystallography to the identification 
of the structure of DNA). As seen in Figure 1, many of the challenges developed through 
this project involved engineering a new solution. For example, sixth graders designed 
structures that would allow us to comfortably live on the moon, after doing some second- 
ary research and science investigations to understand how the lunar day is different than a 
day on Earth. Many engineering lessons did not explicitly invoke the practices of engineer- 
ing design, however. Students were not often asked to specify criteria and constraints of 
successful solutions, or to systematically compare different solutions to arrive at optimal 
designs, making this another area for improvement for similar initiatives in the future. 


Students taking action to make a difference 


The third and final stage of CBL is for students to take action and rise to their challenge. In 
the CBL units designed in this project, most actions had to do either with awareness raising, 
or with conceptualizing and/or creating products or processes to help the environment 

or improve the human condition. In some lessons, teachers gave students some choice 

on how they could “act.” For example, students could try to help plan for, prepare against, 
predict or prevent an earthquake. This resulted in students creating anything from earth- 
quake kits, to blueprints of earthquake proof desks and buildings. In other lessons, teachers 
placed more constraints on what students could do in the action phase of the CBL unit 
(e.g., everyone had to create a 30-second public service announcement about antibiotic 
resistance or to create a container that keeps things cool), but students had freedom on 
several key elements of design and execution. 


Use of open educational resources 


When possible, teachers used OER to ensure that the lessons they created could be shared 
with other teachers without any copyright barriers. While teachers co-created a resource 
bank of existing middle school science OER (Appendix E) and were provided some time to 
explore these resources, most lessons teachers created did not rely substantially on existing 
OER in either design or conception. Teachers relied most heavily on their personal Knowl- 
edge and experience with their students, as well as practical considerations (e.g., taking into 
account the content that needs to be covered when the lesson will be implemented) to 
decide on their lesson topic. Once the lesson topic was decided, it seemed more efficient 
to Google search for relevant materials rather than look through different OER websites on 
the chance that they would contain something relevant to the planned challenge.” 


? Middle school science OER can exist in large extended units (e.g., sample units in OpenSciEd) and in smaller instructional bits (e.g., single 
lessons, rubrics, and activities). If teachers are looking for something very specific to insert into their lesson, the Google strategy is simpler. 
They might have had more incentive to look at OER sites if they were looking for examples of whole learning activities. 
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What felt new and beneficial for teachers 
Student engagement 


The vast majority of project teachers thought students seemed very engaged with their CBL 
lessons, relative to the lessons they usually teach. Teachers thought that students may have 
been engaged by the phenomena, by the ownership of their learning, and/or by having 
their question read aloud and considered by the entire class. 


Trusting students to ask good questions and research them on their own 


Asking students to formulate and pursue questions was a very different approach from 
most teachers’ typical practice. For many teachers, asking students to do more of the 
questioning and thinking worked out better than they had expected, which was a pleasant 
surprise. “l was surprised that the kids came up with some good questions, and that they 
came up with questions that | wanted them to ask,” said one teacher, “And it’s basically 
the questions a teacher would ask, but they actually generated the question.” The project 
required teachers to trust their students in ways they don't typically think to do. Another 
veteran teacher shared: “We're used to ingraining the content with our lessons, so [it feels 
different that we're] relying on the kids to learn the content and trusting that they're going 
to be dedicated enough.” 


Getting students to discover information rather than to receive and practice it 


On a related issue, many teachers pointed out that the challenge based lessons were 
different than their usual practice because students were discovering and constructing 
knowledge themselves in an open-ended environment, rather than “taking notes and 
practicing things.” As several teachers pointed out, this had to do with the sequence of the 
lesson being very different than in most teaching. Appreciating NGSS experts’ advice that 
it is more effective to flip the common practice of “frontloading” students with informa- 
tion instead using activities to motivate obtaining needed information, one eighth-grade 
teacher remarked: 


The other thing that | really learned from this is the information the kids can pick up 
that you do not have to introduce in the beginning ... | did not need to frontload any 
information. | didn't need to frontload vocab. | didn't need to frontload concepts ... 

| didn't really have to tell them that much about MD Olympics or disabilities. They 
figured all of that out on their own. ... [Elven with the prototyping and all that kind of 
stuff, | didn't have ... we didn’t have to sit there and have a lesson on “How do you 
make a lever?" They just figured it out. 


Learning more about students 


The project provided extended opportunities for teachers to observe how students behave 
when they need to figure things out on their own. On the one hand, teachers saw many 
ways that students are capable of guiding their own learning. Teachers observed, for exam- 
ple, that many of their students can productively generate and pursue science questions, 
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reason more deeply than they might have expected, come up with creative ideas, learn 
from failure, use technology effectively, and work together naturally and productively. On 
the other hand, teachers also were able to see new areas in which students struggled. 
Several teachers observed that their students were not used to thinking for themselves, and 
needed more support. An eighth-grade teacher shared: 


They're not used to questioning themselves or questioning what we're doing. Again, 
they're spoon-fed, “Here's this information.” So if they're asking questions, [they won- 
der] “What kind of questions should | ask?” So, | think moving forward, that’s a concept 
that we need to address. 


Other teachers made similar observations about students’ lack of experience in question- 
ing, sensemaking, and intellectual risk-taking. Some observed that many of their students 
didn't know how to conduct secondary research (e.g., they relied on Google Images and 
avoided reading). Others realized their students were not as good as they had assumed 

in applying concepts to solve new problems. In addition, others remarked that students 
were not used to pursuing or trying to address “root causes” of problems. Gaining a better 
understanding of their students’ areas of strength and weakness provided teachers with 
ideas about their future lessons. 


What felt new and beneficial for students 
Real-world learning, independence, and enjoyment 


Many of the challenge based science lessons presented students with authentic and 
relevant challenges and topics. Students explored pressing issues affecting their daily 

lives including global warming and the loss of pollinators. Some students explored topics 
important to their community, such as a challenge to help another classroom with a 
Special Olympics event. As final projects, some of the lessons asked students to create or 
design solutions that a person might use. For example, in a classroom where students were 
exploring the harmful effects of water pollution, one group of students designed a water 
filter. This process of being presented with a real problem and thinking through how to 
address it was a new type of learning for some students. 


Students enjoyed participating in the challenge based science lessons which they saw as 
very different from their normal lessons. For some, it was an opportunity to explore a new 
topic; for others it allowed them to gain experience doing research or developing a solu- 
tion. One student mentioned that what they enjoyed most was being able to lead their own 
learning. This was different from their normal science lessons in that students explored a 
topic they cared about rather than a topic someone else thought they needed to learn. An 
eighth-grade student said: 


| just like how [the teachers] gave us a lot of independence. That was the main part. 
They just gave us a lot of freedom of, “Do what you want, and make sure that you 
actually care for the project.” Usually what we do in science was always like, "We want 
you to learn this. That's why we're giving you this thing to do.” This was mostly for the 
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environment. We actually needed to do this, and it actually mattered, not just for us to 
learn. It actually mattered for, | don’t know, the environment. They did give us a lot of 
responsibility and freedom to make our choices. 


Several also appreciated being able to help others. For example, in one lesson where 
students were challenged to think through how they might help their friends in another 
classroom participate in a Special Olympics event, they designed a prototype to make 
participation in a throwing event easier. Students who participated in this lesson shared that 
it opened up their knowledge about the types of disabilities that exist and their impact on 

a student. It allowed them to practice considering other perspectives and empathy. One 
eighth-grade student shared: 


[| enjoyed the lesson] because | didn't know some of the disabilities that some people 
have. | didn't know what they were, and it was cool learning about them. And the 
challenges they go through. Because if we wouldn't have done that lesson, | would 
have no clue. 


Hands-on; Being like a scientist 


As mentioned, several of the challenge based science lessons gave students the opportu- 
nity to dig into the science topics they were learning firsthand. In one class, students were 
given the challenge of identifying a stream into which to release their class trout. As part of 
this activity, the teachers brought stream samples from multiple sources filled with sedi- 
ment and invertebrates for the students to examine and analyze to inform their decision. 
One eighth-grade student shared that they appreciated having the chance to do this on 
their own rather than directly hearing from the teacher what the conditions of each 


stream were: 


We saw a bunch of tadpoles. | like the hands-on experience, not with the field trip but 
with the lesson before [in the classroom]. [Rather than the teacher] just saying things 
like “we saw this, and this, and this, and the temperature was right, so we're going to” ... 
we got to actually see what was in the water and what the trout would end up living in. 


Students who had the opportunity to do dissections, experiments, simulations, and engi- 
neering design activities expressed similar sentiments of appreciation. 
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Working with others 


Students also appreciated having the opportunity to work with friends in groups. Many 

of the challenge based science lessons provided an opportunity for students to conduct 
research, present, or build prototypes with another person or a group. For some students, 
this difference stood out as a benefit. One seventh-grade student mentioned this opportu- 
nity because it allowed for their group to figure out how they would address their 
challenge together: 


The cool thing was that everyone got to choose whatever they wanted to do. Each 
block, | think, got to sort of make their own question that they had to solve, but it was 
almost the same one for every block. And then we split off into groups, right? So then 
we were able to make our own choices on what we wanted to do as a project. Like 
my group made an app while other groups made an invention to stop all the pollution 
going out into the air. 


Expanding their view of the world 


Through this project, students addressed challenges related to global warming and its ef- 
fect on food production, antibiotic-resistance, and the impact of blue light. These authentic 
complex challenges allowed students to expand their understanding by developing their 
own opinions related to these topics. In one lesson, where students worked towards de- 
veloping solutions for global warming, a seventh-grade student reflected back on learning 
about this topic and realizing how pressing and relevant it is for them today: 


| knew it had an impact, but | didn't realize how big it was until we did this unit. | 
thought it was just something that was on the back burner, just like there but it didn’t 
really impact us that much. 


Areas for further improvement 


Teachers identified several areas they wanted to work on or still had questions about. Many 
of their thoughts for improvement had to do with sequencing and logistics of their lesson, 
especially around how the student questioning can be smoother, or how the project 
requirements could be improved. Several teachers had questions about how to assess 
students’ work. For example, if students are all doing something a little different, how can 
they be fairly assessed? Also, how can the assessment be made so that it doesn’t constrain 
the students, yet at the same time indicates whether or not they've learned important skills 
and ideas? Do (and how do) challenge based lessons help traditional content learning and 
performance on conventional assessments? And finally, while a lot of CBL is about making 
connections to real life, how can a teacher know whether students are in fact making 
those connections? 


Some teachers wondered about how to promote deeper research skills for students, 
including how “to make sure all students are coming back to all the core science ideas 
[after their research]” and “to make sure through their process of research there's a 
depth of understanding.” 
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Section 3. Deeper Learning Opportunities and 
Outcomes in Challenge Based Lessons 


In this section, we discuss evidence on deeper learning opportunities and outcomes 
afforded by the challenge based science lessons. The discussion centers primarily around 
an analysis of 71 samples of middle school science lessons and associated student work. 
Two kinds of lessons were collected in addition to the challenge based lessons. This was a 
“typical” lesson that teachers characterized as representative of what students usually do in 
their classes. The second type of comparison lesson was selected by teachers to represent 
something they had done in their class that they thought was relatively well-aligned with at 
least one of the NGSS science and engineering practices ("NGSS practices-aligned"). Most 
of these comparison lessons were collected during the baseline visit before the project 
started, although some were collected later. Table 4 describes the three types of lessons 
we examined and compared. 


Table 4. Three Types of Lessons the Project Collected and Examined 


Lesson Type 


Typical Lessons that teachers reported as being typical. 
Many lessons involved students obtaining 
information from textbooks or online sources, 
and/or applying terms to demonstrate 
their knowledge. 


NGSS Practices-Aligned Lessons that teachers reported as being relatively 
well-aligned with at least one of the NGSS 
science and engineering practices, or if they were 
not familiar with these, science inquiry. Most 
lessons were either guided labs, inquiry labs (i.e., 
labs where students had to figure out some of 
the process), or lessons where students needed 
to engineer solutions. 


Challenge Based Lessons that teachers created through 
this project. Lessons included guided labs, 
researching and designing solutions, simulations, 
dissections, data collection, and analysis. 
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To identify whether lessons exhibited deeper learning opportunities and outcomes, the 
research team designed rubrics that were used to score the lessons and student work. The 
rubric dimensions represented learning domains that we identified as important to middle 
school science teaching and learning, based on the literature as well as project site visits 
and teacher input. Table 5 provides the set of dimensions for lessons and student work. 


Table 5. Middle School Science Deeper Learning Dimensions Examined in this Report 


Learning opportunity dimensions Student work dimensions 


Activity calls for students to... Student(s) ... 


* engage with real-world phenomena or problems ¢ did/created something valuable to others as 
that connect with their interests and values part of their learning 


¢ guide their own learning of science or ¢ learned through practicing science 
engineering 
* made sound use of the three major stages 


¢ discover principles or effective designs through of the engineering design process 
direct experience 


* critically researched existing information 
* critically research existing information 

* produced a product with effective 
¢ collaborate substantively organization, style, and format 


¢ effectively organize, style, and format 
their communication 


The rubrics went through a multi-stage development process described in detail in lwatani, 
Vang, Romero, and Means (forthcoming). Care was taken to attend only to deeper learning 
factors that are closely related to science learning, and to align expectations with NGSS 
when possible. The goal was for the rubrics to capture skills and dispositions that teachers 
would regard as being central to their science instruction, rather than auxiliary or addition- 
al. Each dimension had five levels (0-4) where generally, 0 indicated absence of the deeper 
learning opportunity or outcome, 1 indicated emergence, 2 indicated partial presence, 3 
indicated solid presence, and 4 indicated a presence that was above what might be expect- 
ed of middle school learning opportunities and outcomes. Examples are provided in Tables 
6 and 7, while the full set of rubrics is provided in Appendices B and C. 


In the summer of 2019, four science teachers and two education researchers used the 
rubrics to score 71 lessons and associated pieces of student work. Each lesson had up to 
two pieces of student work produced in response to the lesson. To support comparing 
“apples-to-apples,” every piece of student work examined was what teachers considered at 
or above expectations for the assignment. In other words, student work that was clearly of 
low quality or incomplete was treated as ineligible for this analysis. Almost all lessons and 
more than 90 percent of the student work samples were scored by at least two scorers, 
with more than two-thirds of the artifacts scored by three or more scorer. Scorers had no 
knowledge of the project goals or research questions, and were not told which lessons 
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were challenge based, typical or practices aligned. Each scorer scored their assigned 
lessons on the six deeper learning opportunity dimensions and five deeper learning student 
work dimensions (listed above in Table 5). In addition, the scorers provided their assessment 
on how well each lesson provided opportunities for students to understand grade-appro- 
priate core ideas in science and/or engineering per the NGSS, and how much the student 
work demonstrated a sound understanding of these disciplinary core ideas (also based 

on the 0-4 point scale, as described further in Appendix D). As a group, the scorers were 
very consistent in their scoring using the deeper learning rubrics, and moderately con- 
sistent in their assessments concerning disciplinary core idea learning opportunities and 
demonstration. 


This section presents the average scores of each deeper learning opportunity and student 
outcome (as seen through student work) dimension across different lessons in a lesson 
type. Appendix A provides more details on the scoring and analysis process, including infor- 
mation about score reliability. 
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Table 6. Example of an Activity Rubric Dimension (full set provided in Appendix B) 


Activity Dimension 1: Activity calls for students to engage with real-world phenomena* or 
problems that connect with their interests and values. 


Activity 

doesn’t involve 
real-world 
phenomena or 
problems (e.g,., 

it asks students 
to learn facts 
ora theory ina 
decontextualized 
way). 


Examples: 


Worksheet on 
Punnett squares. 


Demonstration of 
evaporation and 
condensation 

of water. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity doesn't 
call for students 
to invest much 
time, emotion, 
or thinking 
towards the 
phenomenon 
or problem. 


Examples: 


Reading or 
watching a video 
about the impact 
of deforestation 
and answering 
questions. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity calls 

for students to 
invest time and 
thinking towards 
the phenomenon 
or problem. 


Activity doesn't 
attempt to 
engage students 
emotionally. 


Examples: 


Designing the 
tallest building 
possible. 


Investigating 
how finches 
have evolved 
over time. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity calls for 
students to invest 
time and thinking 
towards the 
phenomenon or 
problem. 


Activity attempts 
to engage 
students 
emotionally by 
connecting with 
their interests 
and values. 


Examples: 


Designing a more 
effective layout 
for the school 
cafeteria. 


Investigating 
the nutritional 
content in chips 
vs vegetables. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity calls for 
students to invest 
time and thinking 
towards the 
phenomenon or 
problem. 


Activity attempts 
to engage 
students 
emotionally, by 
connecting with 
their interests 
and values. 


Learning about 
this topic can 
greatly expand 
students’ 
awareness and 
change the way 
they think, feel, 
or act. 


Examples: 


Designing a more 
effective home 
layout for a local 
resident who is 
paraplegic. 


Investigating 
the nutritional 
content in food 
that students 
want to learn 
more about. 


*Note: "Phenomena are observable events in nature (or our lives) that connect to multiple NGSS disciplinary core ideas, such as Finnish Snow 
Trees or the behavior of bees" (Maltese, n.d.). 
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Table 7. Example of a Student Work Rubric Dimension (full set provided in Appendix C) 


Student Work Dimension 1: Student(s) did/created something potentially valuable to 


others as part of their learning. 


Assignment 
didn't 

ask for 
student(s) 
to create 
something. 


Examples: 


Notes, 
guided labs. 


Student 
product is 
missing, 
incomplete, 
and/or 
misses 

the point. 


Examples: 


A poster 
about 
weather, 
when the 
assignment 
was about 
genetics. 


A product 
was Created, 
but its 
interest/ 
value to 
others is 
questionable 
(e.g., purpose 
is unclear 


or not 
compelling). 


Examples: 


A poster that 
shows the 
stages of 
mitosis. 


A word 
search that 
contains 
genetics 
terms. 


Student 
did/created 
something 
that could be 
of interest/ 
value to 
others. 


But the 
product 

is not 
particularly 
original or 
creative. 

It mainly 
repurposes 
already 
available 
information in 
a simple way. 


Examples: 


Educational 
game or 
poster 

that raises 
awareness 
about 
genetically 
modified 
foods. 


Student did/ 
created 
something 
that could be 
of interest/ 
value to 
others. 


The student's 
work is 
original 

or creative. 


Examples: 


A prototype 
of an app 
that supports 
parents with 
a genetic 
mutation 
making 
decisions 
about having 
a child. 


Student did/ 
created 
something 
that could be 
of interest/ 
value to 
others. 


The student's 
work is 
original or 
creative. 


In addition, 
student 
provided 
evidence that 
either (1) they 
thoughtfully 
reflected on 
strengths and 
weaknesses 
of their 
design and/ 
or (2) their 
product was 
appreciated 
by others. 


Examples: 


The app 
(mentioned 

to the left), 
with a letter of 
support from 
a user, and/ 
or a written 
reflection on 
strengths and 
weaknesses of 
the product. 
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Challenge based science lessons provided more deeper learning opportunities than 
“typical” science lessons did. However, there is still room for the lessons to improve. 


Figure 4 compares the average deeper learning scores across all challenge based science 
lessons (n = 39)° with those of typical science lessons (n = 17). The results are highly 
encouraging with respect to the potential for challenge based science lessons to provide 
deeper learning opportunities to students. On average, challenge based science lessons 
provided substantially more opportunities for: real-world engagement, self-direction, prac- 
tice of science and engineering skills, conducting critical research, substantive collabora- 
tion, and effective communication. Moreover, challenge based lessons did this while being 
just as strong as typical lessons in terms of providing opportunity to learn grade-appropri- 
ate science content.* Hedges’ g, a measure of how large the average score differences are 
in terms of standard deviation units, ranged from 0.88 to 1.75 for the difference between 
challenge based and typical lessons on the various deeper learning rubrics in our analysis. 
This suggests that the deeper learning scores of the challenge based science lessons were 
not just higher, but very much higher on average than scores of typical lessons. 


However, these results also indicate that there is room for improvement in the challenge 
based lessons, especially in increasing opportunities for students to practice science 
investigations and engineering design, and to conduct secondary research more critically. 
Corroborating what the research team observed through classroom visits, the indepen- 
dent scorers who examined the challenge based lessons found that the deeper learning 
opportunities in these two areas, while stronger than the opportunities in typical lessons, 
were still just “emerging” (i.e., a score of approximately 1). So while challenge based lessons 
designed in this project provided significantly “more” opportunities than did typical lessons 
(that on average scored approximately O or a little over), the challenge based science 
lessons can improve much more on this front. 


3 This number is larger than the number of CBL units that were developed through this project because some CBL challenges consisted of 
multiple lessons (e.g., the cell phone jail challenge comprised a lesson where students did research on the nature of cell phone radiation, and 
another lesson where students designed a device that blocks them). Also, if some lessons were implemented differently across classrooms 
(even if they were triggered by the same CBL challenge), they were considered to be different lessons (e.g., one implementation of the 
cell phone jail challenge asked students to build and test a device, while in another, students were asked to draw a blueprint). These were 
considered to be “different” lessons because they were substantively different in what students were asked to do. 


4 The mean scores on science content were 2.03, 2.30, and 2.36 for typical, practices-aligned and challenge based lessons, respectively. There 
was no Statistically significant difference between these scores (Appendix A, Tables A1 and A2). 
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Figure 4. Deeper Learning Opportunities in Challenge Based and Typical Science Lessons 


Hedges’ g = 1.75 


Average rubric score 


Real-world Self-direction Practicing science & Critical secondary Substantive Effective 
engagement engineering research collaboration communication 
IB Typical lessons I Challenge based lessons 
(n = 17) (n = 39) 


Note: All differences displayed are statistically significant (p < .05). See Appendix A, Table A1, A2, and A3a for relevant data tables. 


We also compared deeper learning opportunities provided by the challenge based science 
lessons to those in lessons that teachers provided as examples of “NGSS practices-aligned” 
(Figure 5). The latter lessons were not typical of what teachers usually did in class but did 
represent what they regarded as their best approximation of the kind of teaching called 

for in the NGSS. On average, the challenge based lesson scores were no different than 
those of NGSS practices-aligned lessons in three of the six deeper learning dimensions: 
self-direction, practicing science and engineering, and substantive collaboration. However, 
the challenge based learning lessons scored significantly higher than the NGSS practic- 
es-aligned lessons in the remaining three areas: real-world engagement, critical secondary 
research, and effective communication. 
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Figure 5. Deeper Learning Opportunities in Challenge Based and “NGSS Practices-Aligned” Science Lessons 


Hedges’ g = 1.60 No difference 


No difference No difference 


Average rubric score 


HIP 


Real-world Self-direction Practicing science & Critical secondary Substantive Effective 
engagement Tate at-salare} research collaboration communication 


IB NGSS Practices-aligned lessons [ll Challenge based lessons 
(n = 15) (n = 39) 


Note: Statistical significance level set at p < .05. See Appendix A, Table A1, A2, and A3b for relevant data tables. 


When students in diverse classrooms are provided with opportunities to learn more 
deeply, they seem to do so 


Did students learn more deeply when provided with deeper learning opportunities? To 
address this question, we examined the student work produced in response to lessons with 
high and low scores on deeper learning opportunities. For example, if a lesson called for 
students to do critical research, we would expect the student work to demonstrate that 
students have applied critical research skills. Conversely, if students were not asked to strive 
to communicate their science ideas effectively, we would not expect students’ work to 
demonstrate strong communication skills (though some students might naturally be good 
communicators or have learned communication skills through other classes). 


We found that, as logic would suggest, students’ deeper learning outcomes are positively 
and strongly correlated with opportunities to learn deeply (Table 8). For example, the more 
opportunities a learning activity provided for students to conduct science and engineering 
practices, the more evidence was found of science and engineering design competencies in 
students’ work. 
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Table 8. Correlation Between Deeper Learning Opportunities and Student Outcomes 


Learning Opportunity Student Outcomes (as seen Correlation* 
in work) 


A1. Real-world relevance S1. Did something relevant (of 


interest/value to others) 


A3. Conduct science/ S2. Practiced science 
engineering practices 
S3. Used engineering design 


A4. Conduct critical secondary S4. Conducted critical 
research secondary research 


A6. Communicate effectively $5. Communicated effectively 
Understand disciplinary core Understood disciplinary core 47 
ideas ideas 

) 


* Polychoric correlation using bootstrap estimation. All correlations were greater than O (p < .001). 


When participating in challenge based science lessons, students did more critical 
secondary research and real-world learning 


We saw earlier that CBL lessons provided deeper learning opportunities to a greater extent 
than typical and NGSS practices-aligned lessons did on several dimensions. Did these 
differences correspond to differences in deeper learning outcomes for students? Figures 
6 and 7 show how student work scores for challenge based lessons compared to those 
for typical lessons and NGSS practices-aligned lessons, respectively. The data suggest 
that students were more likely to have done something that could be of interest/value to 
others (e.g., the community or society), and were more likely to have conducted critical 
secondary research through the challenge based lessons, relative to both typical and 
NGSS practices-aligned lessons. In addition, in the challenge based lessons, relative to the 
practices-aligned lessons, students were less likely to have used NGSS science practices 
but more likely to have communicated effectively (see Figure 7). 


These results are not surprising; they corroborate teachers’, students’, and researchers’ 
reports that in challenge based lessons, students engaged more with the real-world, 

did more secondary research, and communicated more often and more effectively. The 
student work scores generally did not reach the higher levels of the rubric in part because, 
as we saw above, most lessons did not call for students to reach the highest levels. This 

is understandable, especially when students are still inexperienced in guiding their own 
learning (as was the case with the great majority of students in this project). 
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Figure 6. Challenge Based Science Student Outcomes Compared to “Typical” Science Student Outcomes 
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Note: Statistical significance level set at p < .05. See Appendix A, Table A4, A5, and Aa for relevant data tables. 


Figure 7. Challenge Based Science Student Outcomes Compared to “NGSS Practices-Aligned” Science Student Outcomes 
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Note: Statistical significance level set at p < .05. See Appendix A, Table A4, A5, and A6éb for relevant data tables. 
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Exploration of factors potentially influencing the quality of deeper learning 
opportunities in challenge based science lessons 


We conducted exploratory analyses of several variables that could be expected to produce 
variation in deeper learning rubric scores. First, we examined descriptive statistics and 
conducted an analysis of variance on the overall score to determine whether the challenge 
based lessons produced and implemented after the first teacher workshop differed signifi- 
cantly from those produced after the second teacher workshop. We found that there was 
no significant difference between the first and second rounds of challenge based lessons, 
and that the variance was between challenge based and comparison lessons. Next we 
contrasted deeper learning ratings for the lessons developed by teachers in Title | schools 
(n=8) versus those in non-Title | schools (n=10). Again, the differences were not statistically 
significant (see Appendix A, Tables A7, A8, and A11). This suggests that challenge based 
science lessons provided comparable deeper learning opportunities across the two types 
of schools. It should be noted, however, that the relatively small size of the lesson subsam- 
ples limited our power to detect differences through these exploratory analyses. 
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Section 4. Lessons for the Field 


This section summarizes lessons learned and advice from teachers and students to ed- 
ucation practitioners and the field of teacher education more generally. In addition, we 
summarize insights from the Digital Promise research team. 


Advice From Teachers 
Letting go of control is hard, but well worth it. 


After creating and implementing challenge based science lessons, many of the project 
teachers said their key takeaway was that it was difficult, yet worthwhile, to provide stu- 
dents with some control over their learning. One shared that despite her initial worry, “It’s 
okay to give them full rein, and in regards to the learning process, [allow students] to come 
up with their own questions and researching and creating.” Others similarly said their main 
advice or takeaway was that teachers need to “learn how to let go,” “be able to let go of the 
reins,” “allow for student discovery,” or “don't always have to be the one presenting.” 


Many teachers felt that providing students with opportunities to direct their own learning 
was rewarding because it gets all students in the class to participate in a deeper and more 
personal way (see Boxes 1 and 2 for examples). Several teachers reported anecdotes about 
how they were pleasantly surprised to see students who had been disengaged in class 
throughout the school year actively participate in their challenge based lessons. Some 
teachers also noticed that their higher-achieving students were challenged in a new and 
important way, as they needed to figure things out for themselves rather than memorize 
facts and procedures. Letting students work on their own projects turned out to be quite 
enjoyable for some teachers, allowing them to discover new aspects of students. As one 
teacher put it: “I just really like the fact that they get to kind of do their own thing ... it was 
kind of neat to see what they came up with." 


Teachers from three of the five participating schools, all with lead roles in their schools’ 
science curriculum, reported in their interviews that they now really understand the value 
of students trying to figure things out for themselves. One teacher shared how his under- 
standing flipped, and that he now sees the hands-on exploration part as being where the 
student learning really needs to occur, rather than it serving as simply a “fun” phase that can 
be referenced in the future for (didactic) learning. Another teacher saw that even her top 
students were not able to transfer the concepts she thought they should have learned from 
all their labs and notes, so decided to take two additional class periods to have students 
solve a mini-challenge about earthquakes. “It takes longer [for them to figure it out on their 
own], but | think they learn better that way.” Yet another teacher shared that her “ah-ha” 
moment was when her class seemed to intuitively derive the notion of “balanced force" 
after their exploratory investigation using helium balloons, even when she did not teach 
them any of the vocabulary in advance. The eighth-grade teacher concluded: 
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| think a lot of teachers would say, “Well, why can't | just stand in front of the classroom 
and tell them what this is?” But | think walking away long term, they’re going to keep 
that knowledge a little longer having had to figure that out themselves instead of me 
just dictate it to them. 


|sXoy a w-Na-t(e] aldabre | ¢-Lel-m-voll-) aX-mc-t-Lol al-) om = (-rol dela ela 


Cot a-1 (Yate [=m o-bX-10 Yel (-Varec-m(-r-laal are) 


“It's definitely challenging to go outside the box of what you're used to doing. 

But | think it’s rewarding. | think you see the light bulbs going off in your kids. You 
see that they're more engaged. ... [And because | did the lesson with all my class 
periods, rather than just my strongest], | saw a lot more of the kids that don't 
participate, or don't do anything. | did see a lot more of them kind of turning ona 
little bit. Even if they didn’t complete the whole entire task, there were times and 
bits and pieces of where they were like, ‘Oh, I’m going to do this part. I'm going 

to do this part of it.’ ... And when you teach to a test like so many of us are doing, 
unfortunately, you start to see your kids in a different way. Some kids that you 
think are never going to do anything, they actually show you, you know, ‘lam 
interested in science. | just don’t like the way that we do it every day.’ So | think it’s 
kind of eye opening too when you see some of the kids in a different way when 
they're creating, and doing, and investigating, and researching on their own versus 
what you tell them to do. And worksheets and notes. So | think it gives you, like 
an insight to what they might be like outside of that. | think it’s rewarding in that 
manner.” (Eighth-grade veteran science teacher, after creating and implementing 
two challenge based science lessons) 


“You know, | reflected a lot on just my teaching. | learned more of the content. The 
teaching style was definitely different, for the better. | think it’s something that can 
work in the future. Will | do it again? | definitely will do the [challenge based science] 
unit again, if | could get the standards for next year, because | know we're re-rout- 
ing. But | would definitely do that again. | don’t know. | felt that it ... | felt almost 

this type of learning does make the connection a little stronger with them. It allows 
me to sit with groups and move around a lot, where I’m not just up instructing. | 

did notice that—where | can move and sit. Like, | would just sit with groups and 

talk to them. There was more of an individual instruction piece as well. This type of 
instruction allows for that. It does. | felt like | had more time in that 80-minute block 
where | can meet with individual groups, whereas if I’m doing direct instruction | 
feel like I'm up there constantly and not able to meet. Whereas this time everyone's 
working, so | can scoot around for five or 10 minutes per group ... Yeah. That is true. 
That is true! Yeah. | didn't think about that until just now. That's true.” (Seventh-grade 
veteran science teacher, after creating and implementing two challenge based 
science lessons) 
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Hardest part is the “engage” and facilitating student questioning, but practice helps. 


Teachers thought that getting students to “engage” in their challenge was difficult because 
it could be hard to find a topic that most students would like. It is also not easy to facilitate 
middle schoolers’ varying opinions and thoughts in a productive direction. Middle school 
teachers receive training in many areas, but typically not in facilitative management and 
project supervision. So understandably, many teachers who were facilitating student 
questioning for the first time (more than three-quarters of the participating teachers) felt 
somewhat nervous about the process. Some worried about whether they could be quick 
enough on their feet to respond to unanticipated questions (especially when they were 
being observed by researchers!), and some about worried whether students could come 
up with questions, move towards a productive direction, or carry things out on their own. 
Despite such initial worries, teachers experienced success, often to their surprise: 


| was most worried about organizing all the sticky notes [with individual student 
questions triggered by the phenomenon] and stuff and I’m not very good at somebody 
coming out saying something me just coming up with something real quick. You 
know, I'm not witty like that. | Kind of have to think about things. And that sticky note 
totally puts you on the spot because you have no idea what they're going to ask, and 

| surprised myself just by going with it ... and being able to organize it. | was proud of 
myself...| learned when | am able to just, you know, spurt some things out without 
having to think about it. (Seventh-grade science teacher, after implementing first 
challenge based lesson) 


Teachers who were nervous about facilitation agreed that it gets much easier with practice. 
It gets easier throughout the day and the second challenge based lesson was much easier 
than the first. One teacher shared: “My first period class, they get all my stumbles. And then 
by the time | go to the afternoon, it’s a lot better.” 


Even when students are in control, you can still guide where they're going. 


Participating teachers had reassuring advice for colleagues who might worry about main- 
taining classroom control when students drive their own learning. One veteran teacher 
said that it was very important for her and other teachers to learn that they “just have to go 
with the flow and lose control.” But this doesn't mean letting students do whatever they 
want. She said if students go in a direction she likes, she can “go with it,” and the learning 
is that much more powerful. Indeed, she was surprised that sometimes students came 

up with good ideas that she had not thought of. However, if she doesn't like where her 
students are going, she can (and should) redirect them the next day, and this is not such an 
onerous task. Similarly, other teachers used metaphors like wind and sailboat, or a stream 
and a canoe to describe their relationship with students during challenge based lessons. 
With students steering their learning, the teacher is like the wind or the stream that heavily 
influences the driver's direction, and on which the driver depends. Teachers used a variety 
of methods to guide students without being prescriptive, including highlighting student 
questions or comments that were particularly aligned with science, creating check-ins to 
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catch students before they got too lost, and adding their own questions in ways that would 
nudge students towards more productive directions. 


The ideal balance between student-directedness and teacher-directedness can be tricky 
to achieve and needs continuous refinement. 


Teachers continuously wrestled with the balance of how much freedom to provide their 
students, and how much they should guide or scaffold. Should they specify what product 
students should make, and if so, how specifically? To what extent should they shape the 
questions students pursue or how much research they do? If the teacher keeps things 
open-ended, how should students be assessed? In some cases, keeping things open-end- 
ed worked much better than teachers expected. For example, one teacher provided fairly 
detailed guidelines for secondary research in some classes, asking students to fill in a 
worksheet that specified the number of questions and providing boxes where students 
would fill out answers to their questions. Seeing that this structure seemed to be constrain- 
ing students, she switched her approach in her afternoon classes to ask students simply to 
research what they thought was important to know about the topic, without providing the 
worksheet. She was surprised that students in the afternoon classes seemed more engaged 
with the process and conducted more thorough research than did students in the 


morning classes. 


On the other hand, teachers also observed that students needed some amount of struc- 
ture, especially when they were new to identifying and solving big challenges. Many teach- 
ers found a balance by having some standard requirements. Examples of the requirements 
teachers set within challenge based learning units included requiring students to research a 
core set of questions in addition to their own, stipulating some investigation activities, and 
giving some specific requirements for final products. 


CBL units might be a sustainable alternative to adopting a full project-based learning 
model where everything has to be project based. 


Two of the participating teachers had much less to learn about student-centered in- 
struction than the other teachers did because they were fully trained in project-based 
learning (PBL) and had been implementing it daily for several years. While challenge based 
learning didn't feel new to these teachers, they appreciated the learning curve that other 
teachers were experiencing. One commented that while her personal preference would 

be to go “full PBL,” she understands that's neither the only way—nor necessarily the most 
appropriate way—to engage students, especially in schools with high staff turnover and 
limited resources. It takes many resources to train teachers to do PBL year-round, and not 
all teachers are comfortable with this pedagogy (which she believed to be “completely 
legitimate’). She'd also heard in PBL leadership trainings that it’s unfortunately common for 
PBL schools to revert to “traditional” curriculum and instruction because of staff turnover 
and inability to maintain the required intensive professional development. She wondered 
whether incorporation of shorter-term CBL units, as teachers did in this project, might be a 
sustainable alternative to providing students with deeper thinking and inquiry opportunities, 
perhaps serving as a gateway experience leading to “full-on” PBL or CBL. 
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This was really hard. It would have helped to have examples. 


One teacher seemed to represent the voice of well over half of the teachers when she said 
that the professional learning experience was “really hard” because she had to learn so 
many new things, it took a lot of time, it was a lot of work, it pushed her out of her comfort 
zone, and everything was so “experimental.” Including the time in project workshops, 
teachers devoted at least three days to developing each lesson, and some teachers spent 
additional time on their own. With no prior examples of CBL units aligned to the NGSS to 
work from, teachers had to rely largely on their imaginations, experience, and an evolving 
project lesson template. Some teams went through several major revisions of their CBL 
units before feeling they were ready to implement with students. While all participating 
teachers thought that their hard work was ultimately worthwhile (e.g., one teacher 
described her journey as “hesitant —> frustrated —> enlightened’), they also thought that 
teachers new to the process would be greatly helped by having strong examples of NGSS- 
aligned CBL units. 


Advice From Students 
Try to make the topics relevant to us. 


In focus groups, students frequently shared that they wanted their teachers to continue 
doing the challenge based science lessons, as they were fun, hands-on, and meaningful. 
When students weren't as enthusiastic about their challenge based science lesson (a small 
minority of students), they tended to say it was because the topic did not feel relevant 

to their own lives (e.g., “[Our impact on the environment] is not something your average 
seventh grader would be talking to their friends about"; “[Teachers should give us] more 
experiments, which is like going to help us in our real life. Like real, everyday life.”) 


Across all student focus groups, however, there was no strong consensus among students 
on what topics were interesting or relevant. Interestingly (given prevalent assumptions 
about middle schoolers’ fascination with disaster), the CBL lessons on natural and hu- 
man-made disasters tended to receive less enthusiastic comments than did engineering 
lessons. To researchers, it seemed that the specific topic didn't matter as much as what 
and how much students felt they learned from the unit. When students felt they learned a 
lot about things they didn't know before (e.g., about specific pollinators/species, antibiotic 
resistance, blue light), they tended to regard the topic as being extremely relevant and in- 
teresting. When students felt they already knew a lot about the topic (e.g., negative impacts 
of natural disasters; how humans are polluting the environment), they tended to describe 
those topics as important but not as interesting. 


It's hard if we don't have enough time to do our projects. 


Students had few negative things to say about the challenge based lessons they experi- 

enced, but several did express a wish for more time to focus on their final projects. They 
felt that the emphasis was on the earlier stages of investigating their challenge and there 
was too little time for their final project. In some lessons, students spent two to four days 
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investigating and conducting research, then only one to two days on the “act” portion of 
CBL. This sometimes caused students, especially the higher-achieving students, to address 
the “challenge” in the easiest way they could get away with (e.g., making educational 
slideshows or quizzes that just summarized information they learned rather than building 
on it). 


It can get boring to just read stuff on the internet, to learn what we already (think we) 
know about, or to hear the same presentation over and over. 


While the challenge based science lessons attempted to engage students in ways different 
than their normal lessons, there were aspects that some students found boring or frustrat- 
ing that impacted their experience. Some students found that many times they were just 
researching on the internet for answers related to their topic or for research to create their 
prototype. They preferred doing things that were “hands-on,” like doing experiments and 
building things. 


Another aspect that some students disliked was listening to many presentations on the 
same topic. Some teachers had students present what they learned from their research to 
the class, and students found that many of their peers were presenting the same findings 
since they all had similar questions. These are issues common in teachers’ first attempts at 
CBL and could be addressed in the design of project flow and student grouping strategies 
for future challenge based learning units. 


Researcher Insights 


Student-centered, standards-aligned science teaching requires many skills, so it is 
important to create rich learning ecosystems where these can be cultivated and devel- 
oped as needed. 


It is very difficult to teach middle school science in a student-centered way (whether 
through CBL, PBL, or otherwise), and in ways aligned with NGSS. This is because such 
teaching requires teachers to have all of the following: 


1. Reasonable familiarity with NGSS 


2. Strong grasp of science content knowledge, science and engineering practices, and 
the nature of scientific investigation and engineering design 


3. Comfort with the idea of student-centered learning 
4. Awareness of students’ role in the learning process, and ability to work with that 


5. Strategies to facilitate student-centered learning, including facilitation skills 
and pacing 


6. Having a bank of activities/lessons/investigations that facilitate student-centered 
learning (to draw upon for the “investigate” phase of CBL) 
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7. Experience with successfully implementing student-centered learning 
8. Some immunity/resistance to pressures from testing 


Each of these is difficult to achieve on its own, with the implication that “making stu- 
dent-centered learning happen" in a single classroom or school in a short amount of time 
is not realistic. A broader system of support appears to be needed. Indeed, this project's 
success seemed to rely heavily on teachers, district/school leaders, and Digital Promise 
staff coming together with a shared goal, along with combining their resources and 
talents. Figure 8 depicts assets that each of these groups brought to the project that were 
important for teachers to be able to create and implement the lessons as successfully and 
enthusiastically as they did. 


Figure 8. What Teachers, District/School Leaders and Digital Promise Brought to the Table that 
Contributed to the Project's Success 
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The combination of these assets led to the following conditions that research scholarship 
considers to be essential for transformative professional learning (Box 3). 


Building upon teachers’ knowledge and expertise 


First, the professional development involved and relied upon teachers’ extensive teaching 
experiences that allowed them to create lessons that worked well for their students and 
with their teaching style. Teachers were able to implement their collaboratively developed 
lesson designs with high fidelity. Teachers’ collective knowledge and experience was also 
important in enabling them to clearly identify key takeaways for themselves as they tried 
new approaches throughout the school year. 


Professionally relevant, affective learning towards value-laden goals 


On a related point, the teachers’ learning experiences were professionally relevant, and 
engaged teachers emotionally as well as cognitively. This engagement was fostered by 
teachers’ identification with the values inherent in the project's focus on helping students 
identify and act on things that are important to them. The possibility of helping students in 
an emotionally compelling way seemed to motivate several teachers to try something new, 
even if they had initial doubts about the outcome. 


The professional relevance came from the project deliberately focusing on goals that are 
important for middle school science teachers (deeper, self-directed learning in science), 
and—as best as possible—avoiding cramming in goals that are nice-to-have but ultimately 
less important (e.g., promoting a certain reading or collaboration strategy; requiring all 
lessons topics to be about local phenomena). The CBL framework encouraged teachers 

to provide students with greater substantive control over their own learning to a greater 
extent than called for in the NGSS. The NGSS, however, provided practices through which 
students could engage in authentic science learning, helping to narrow and define what 
students could choose to do with the agency that CBL gave them. The combination of CBL 
and NGSS appeared to be fruitful. 


Intense and experiential learning 


Furthermore, the process through which teachers learned how to design and implement 
challenge based learning activities was intense and experiential. Teachers learned not 
from lectures and presentations of information, but from experiencing the CBL process 
as learners (which many teachers reported as being extremely helpful) and from getting 
student feedback when they tried out their first CBL activities. 


Continuous opportunities for critical reflection 


The project provided continuous and natural opportunities for critical reflection. Teachers 
reflected on their teaching and learning as they designed and implemented their two 
lessons. Release time for co-design supported teachers in having multiple conversations 
about their student learning and engagement with their colleagues, which many found to 
be extremely helpful. Digital Promise provided rich opportunities for critical reflection and 


42 | Deepening Science Engagement with Challenge Based Learning: Research Report Digital Promise 


dialogue through the design workshops and research interviews. The program's focus on 
CBL focused the discussions and reflections around student engagement and student-cen- 
tered learning. 


Authentic relationships 


Finally, authentic relationships were forged among all parties involved (teachers, school/ 
district leaders, and Digital Promise). Teachers got to Know one another better, especially 
if they worked on the same lesson. District and school leaders supported teachers’ 
risk-taking in a number of important ways, including finding release time and resources, 
expressing support and appreciation, and providing instructional leadership. Some leaders 
rolled up their sleeves and worked on creating a CBL lesson right alongside their teacher 
teams. Teachers reported that these concrete expressions of support from their leaders 
were essential to their willingness to move outside of their comfort zone. Leaders likewise 
reported appreciating their teachers’ dedication and willingness to do something new and 
different to support their students and district. Digital Promise also came to understand 
and appreciate each teacher's and school's interests and strengths, which helped shape the 
professional learning and research activities as the project matured. 


Box 3. What makes learning transformative for adults? 


Some learning experiences can be extremely impactful, such as in cultural ex- 
change programs where people with very different values and beliefs learn deeply 
from one another, or when medical students get to know patients and families 
under palliative care. Adult learning scholars have conceptualized such “transfor- 
mative” learning experiences in a number of ways, thinking about it as: 


Large shifts in learners’ cognitive assumptions (e.g., Mezirow, 2000); 
¢ Increased integration of one’s inner and outer worlds (e.g., Dirkx, 2012); 
¢ Gaining new ways to see and know the world (e.g., Kagan, 2000); or 


¢ Becoming empowered with the critical consciousness needed to identify and 
act against structures of oppression (Freire, 1970/2018). 


After closely reviewing more than 200 studies on transformative learning, Hoggan 
(2016) defined it as “processes that result in significant and irreversible changes in 
the way a person experiences, conceptualizes, and interacts with the world” (p. 71). 
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While there may be many ways for learning to be transformative, there is broad 
consensus on the characteristics of learning experiences that tend to facilitate 
powerful shifts in learners’ views, experiences, and interactions in the world (Taylor, 
2006). The characteristics are interrelated and mutually reinforcing. The first is rich 
individual experiences, which include experiences in the classroom as well as life 
experiences that learners bring to the classroom. Learning experiences tend to 

be more transformative if the content is value-laden, the activities are intense and 
experiential, and the participants have greater life experience from which to draw 
and react to the new experience. 


Critical reflection is another important feature. Learning is more likely to be 
impactful if learners reflect on what and how they're learning, and perhaps most 
importantly gain insight on why they learned or realized something they hadn't 
before (Mezirow, 1991; 2000). This type of reflection is facilitated through en- 
gagement in dialogue with themselves and others. The nature of dialogue that 
best facilitates insight and change in values and behaviors is not overly critical or 
analytical, but rather one of a supportive and personal nature. Through dialogue, 
learners can grapple with making sense of what they see and feel, and gain 
feedback and affirmation for changing their beliefs, actions, or values. As such, the 
learning experience should have a holistic orientation to teaching, which refers 
to incorporation of ways of knowing besides the rational “analyze-think-change". 
In particular, affective learning, which utilizes a “see-feel-change” sequence, has 
been considered an important and much more effective facilitator of change than 
a purely rational sequence (Brown, 2006). 


Perhaps not surprisingly then, facilitators who hope for shifts in learners’ core 
beliefs, values and behaviors, should be aware of the contexts in which learners 
operate, and forge authentic relationships with them. We tend to form beliefs, 
values and behaviors to help us survive in our world. If someone requests that we 
relinquish some of our time-tested survival strategies but either does not seem to 
care about our well-being or has no understanding of our environment and needs, 
why would we listen? 
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Section 5. Next Steps 


We have shared evidence that the challenge based science learning project led teachers 
to try dramatically student-centered science instruction, which increased deeper learning 
opportunities for middle school students and led to measurable improvements in key 
aspects of student work. Where does all of this leave us? What are productive next steps 
towards great science learning for middle school students? In this final section of the 
report, we derive implications from our work for current practice and for future avenues of 
research and development. 


What are the implications for the design of professional learning activities for teachers? 


Although the science teachers in this project all came from districts that were implement- 
ing the Next Generation Science Standards (NGSS), only a few of them had a deep famil- 
iarity with the standards and understanding of the concept of “three-dimensional learning.” 
Hence, the learning workshops had to support learning about the standards as well as 
learning about Challenge Based Learning (CBL) deeply enough to be able to represent both 
in original designs for student learning activities. 


As described in Section 4, the veteran teachers participating in this project viewed the 
professional learning activities as very challenging, but at the same time extremely worth- 
while. The professional learning activities embodied the qualities of effective teacher 
professional development identified in research literature (Darling-Hammond, Hyler, & 
Gardner, 2017). They were of sustained duration, occurring over an eight-month period 
and involving more than four days of in-person workshops, at least two additional days of 
lesson co-design time, 10 days of implementation, and many additional hours of reflections 
(on their own, with colleagues, and with the instructional design facilitator and researcher) 
about how the lessons went. The professional learning activities were content focused, 
customized to the Next Generation Science Standards rather than presenting CBL in a 
domain-agnostic way. They emphasized active learning, with the vast majority of time 
spent in co-design activities rather than in receiving information. Professional learning 

was structured as a collaboration in a work context, with teachers designing with other 
teachers from their school or district. It featured modeling of the target practice, with the 
training facilitator using CBL phases and moves to impart NGSS-related concepts. The 
instructional design facilitator, the project researcher, and district/school leaders (in several 
districts) provided coaching and expert support. Finally, the two rounds of lesson design 
with implementation in between were structured intentionally to provide for feedback 

and reflection. 


In addition to incorporating these widely acknowledged best practices, the professional 
learning in this project appeared to benefit from several additional features. The profession- 
al learning called for teachers to make a dramatic change in their instructional approach, 
but only as a “try it” for a relatively brief period of time (first a lesson of a few days and then 
one of approximately one week). This approach made the change low risk compared to 
adopting an entirely new curriculum or new way of teaching for the entire year. Risk was 
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further reduced by having the effort endorsed by administrators in their own districts who 
assured teachers that they would be held harmless with respect to any district prescriptions 
around methods or accountability requirements in tension with the CBL units. Teacher 
learning in this project also benefited also from an alignment of values: CBL’s emphasis 

on engaging students’ hearts and minds appeared to resonate strongly with participating 
teachers. Finally, sustained involvement in this project's teacher learning activities was 
motivated by visible results: The student response to CBL activities was so positive that 
their teachers were motivated to learn and do more. 


How can the challenge based science lessons developed through this project be made 
stronger, with deeper learning opportunities and outcomes for students? 


The challenge based activities that teachers designed and developed through this project 
were consistently superior to teachers’ typical science lessons in terms of inviting the 
aspects of deeper learning embodied in our learning activity rubrics. On some of the 
dimensions, the challenge based lessons even exceeded the lessons the teachers regarded 
as those best-aligned to NGSS practices. But that is not to say that teachers’ first attempts 
at designing NGSS-aligned, challenge based learning activities were completely successful. 
In terms of rubric levels, most of the CBL lessons provided some opportunities for deeper 
learning but fell short of the top rating of fully realized opportunities to acquire deeper 
learning competencies. This result is not unexpected. Teachers need time and ample prac- 
tice opportunities to learn how to create high-quality, student-centered activities around 
scientific phenomena and investigations. The joint design activities and feedback from 
other teachers built into this project gave teachers the confidence to try something new, 
but a stronger set of supports could be provided. Teachers wanted examples of strong, 
NGSS-aligned, challenge based learning activities, and we are only now at a place where 
we can provide examples of activities at different levels of quality on the rubrics. Further 
scaffolding of teachers’ activity design work could be provided by having a set of templates 
that build in NGSS practices and the phases of CBL while allowing for customization to 
individual teachers’ curriculum, interests, or local context. (This idea will be explored 
further below.) 


The influence of the teacher co-design activities on the quality of student work was more 
modest than that on the nature of the enacted science learning activities. This is to be 
expected since the professional learning activities affected the resulting CBL unit designs 
directly while the influence on student work was indirect (mediated by the CBL units). To 
obtain larger, more consistent impacts on student work, we would want to have students 
exposed to a series of challenge based science learning activities in a sequence that builds 
up students’ understanding of what it means to pose a good question about a science 
phenomenon, how to design an investigation to address a question, the components of the 
engineering design process, and so on. Making CBL a larger part of science courses would 
be expected to increase student demonstration of the various dimensions of 

deeper learning. 
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What are the implications for scaling high-quality professional learning and NGSS- 
aligned challenge based learning? 


The collaborative design work with peers, instructional designers, and researchers in this 
project appeared to give science teachers a deeper understanding of both CBL and NGSS 
principles than typical teacher professional development would. But it consumed consid- 
erably more resources, with the main cost being teachers’ cognitive and emotional efforts, 
including time they had to put in outside of their regular work day to develop these new 
lessons. The level of resources required for these learning activities will not be available 
for the majority of teachers in the majority of schools. Scaling up NGSS-aligned challenge 
based learning will require finding less costly ways to support teacher learning and conse- 
quential changes in practice. 


The question is whether it would be possible to obtain comparably deep understanding 

of CBL and NGSS while also motivating teachers to do this kind of instruction, without 
requiring every teacher to participate in designing learning activities from scratch. In other 
words, to what extent was teacher creation of new learning activities a necessary com- 
ponent of this work? Would the same benefits result if teachers used the challenge based 
science units developed by others? Or would a “Goldilocks strategy” of medium-intensive 
professional development—teachers adapting existing NGSS-aligned, challenge based 
learning activities for their specific students while also piloting the modified units—provide 
the best results? 


How can OER support instructional transformation among K-12 teachers? 


While teachers in this project co-created a resource bank of existing open educational 
resources (OER) with Digital Promise staff and were given time to explore these resources, 
most created the resources needed for their CBL units from scratch and incorporated 
existing resources they obtained using general search engines rather than the OER re- 
source bank. Participating teachers were not OER evangelists, and OER use was not the 
motivation for their activities. Nor were project teachers’ districts placing a heavy emphasis 
on OER. The project's original expectation was that teachers’ co-designed learning activ- 
ities would become OER for others to use. Early in the project, though, we realized that 

an infrastructure—to provide quality assurance and coherence across the various learning 
activities—would be needed to make this realistic. 


There is not an obvious pay-off for K-12 teachers to use OER over other materials they 
can access freely and/or adopt through fair use. By implication, OER is more likely to be 
used in K-12 settings when there is a district initiative to adopt OER as a textbook or whole 
curriculum replacement. OER is also not likely going to be a catalyst for radical improve- 
ment in K-12 instruction. In addition, OER “bits” are not likely to inspire teachers to change 
their practice in significant ways. However, in the context of a broader initiative around 
instructional improvement, we believe OER can play a supportive role. One area where we 
see promise is in having templates or “shells” for powerful learning activities in science (or 
another domain of focus). These can be in the public domain and designed in a way that 
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makes adaptation to local circumstances and student interests easy to do. For example, 

a unit built around the challenge of providing a supply of potable water in situations with 
water contamination could offer a set of OER that allow teachers to choose and modify 
alternative contamination scenarios depending on their locale and their curriculum. Despite 
differences in surface features, all of the scenarios would use a common unit structure and 
feature parallel activities and assessments for science and engineering practices applied to 
investigating sources of contamination and testing alternative water purification strategies. 


The feasibility of this idea of customizable OER learning activity templates could be tested 
in concert with investigating more cost-effective approaches to teacher professional 
learning, as described above. The level of teacher engagement, changes in teacher prac- 
tice, and quality of student work could be contrasted for (1) teachers designing their own 
NGSS-aligned CBL units; (2) teachers using customizable OER templates for NGSS-aligned 
CBL units; and (3) teachers using NGSS-aligned CBL units designed by other teachers. 


Final Thoughts... 


It is important to put the work described in this report in the broader context of science 
instruction in the United States. A recent survey of a nationally representative sample of 
science teachers indicated that despite national efforts to promote the Next Generation 
Science Standards, less than half of all middle school science teachers emphasize learning 
how to do science in their classrooms and less than 10 percent emphasize learning how to 
do engineering (Banilower et al., 2018). The reliance on teacher-directed transmission of 
science concepts to the neglect of other pedagogies and aspects of science is not going 
to change absent major systemic efforts and tools to support science pedagogies that 

few teachers have experienced themselves. Sporadic half-day professional development 
sessions are not going to be sufficient to bring about the needed transformation. 


The experiences teachers had through this project were a major departure from typical 
teacher professional development in terms not only in terms of intensity, but also in terms 
of learner-centeredness. This characteristic was the main reason teachers found the expe- 
rience so much more relevant than other professional development they had received. Just 
as the teachers were asking their students to rise to a challenge, the professional learning in 
this project asked teachers to rise to the challenge of designing and implementing stu- 
dent-centered science learning. While not all professional development can (or necessarily 
should) be so extended and participant centered, our intuition is that initiatives that encour- 
age teachers to radically change their instruction to a more student-centered approach will 
always need a significant experiential component. Empirical testing is still needed to learn 
how much of these experiences will be sufficient for different kinds of teachers. 
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Appendix A. Methodological and Analysis Details 
About the Project Participants and Setting 


Eighteen science teachers from three districts participated in the project and study. 

They were a highly experienced group with an average of 16 years teaching, with strong 
classroom management skills, and good rapport with their students. All participants had 
at least two years of teaching experience; just three of the 18 teachers had fewer than five 
years of experience, and the remaining teachers had eight to over 30 years of experience. 
Approximately two-thirds of the teachers were female, and most were White. Many said 
they had been implementing "hands-on" and/or inquiry-based science lessons aligned 
with their state content standards. Approximately one-third of the teachers had received 
extensive training on NGSS; the rest had little or no familiarity with the science standards. 


Participating teachers taught in five socioeconomically diverse public middle schools serv- 
ing grades six through eight in three U.S. districts. Eight teachers taught in Title | schools 
(two schools), with one school having high proportions of White and Black students, and 
the other predominantly Southeast Asian and Hispanic students. The other teachers taught 
in relatively affluent schools (three schools) where the student population was either 
predominantly White, predominantly Southeast Asian, or predominantly White and South 
Asian. Enrollment in the teachers schools was approximately 450 per grade, except in one 
school with approximately 250 students per grade. 


Districts were recruited through a national network of forward-thinking superintendents 
(Digital Promise’s League of Innovative Schools). District partners were selected from 
those that expressed interest in the project on the basis of their capacity to develop and 
implement challenge based, NGSS-aligned curricula. Racial and economic diversity in their 
student demographic was also considered. Two of the districts were small, with only one 
middle school. The third district was mid-sized with three middle schools. One district was 
in a state that had adopted NGSS since its inception, while the other two districts were in 
states that were in the process of changing their state standards to better align with NGSS. 
The leaders who participated in the project included a curriculum coordinator/director, 
assistant superintendents, and a school principal, most of whom had prior training and 
experience with science education and/or student-centered learning. 


Data and Analysis 
Two researchers collected the following data throughout SY 2018-19: 


¢ Teacher interview audio and transcripts (three interviews per teacher, each approxi- 
mately one hour long) 


¢ Classroom observations notes (three waves per teacher, each wave consisting of one 
to three class periods) 


¢ Student focus group audio and transcripts (two waves per teacher; four to eight 
students per group, each approximately 40 minutes) 
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¢ Notes and surveys from two two-day curriculum design workshops 


¢ Notes on informal conversations and correspondences with teachers and school and 
district leaders 


« Examples of science lessons and student work (additional to the challenge based 
lessons, teachers provided samples of “typical” and “NGSS-alignea” activities) 


Teacher interviews were semi-structured and were conducted at the beginning of the 
project (baseline, late fall), after the first curricular design workshop and pilot of their first 
lesson (midline, early spring), and after the second curricular design workshop and pilot of 
their second lesson (endline, late spring). The questions focused on teacher perceptions 
of the challenge based lesson design, implementation, and outcomes, including lessons 
learned. Interview transcripts were summarized for each teacher, and themes were identi- 
fied for each wave as well as longitudinally. 


Student focus groups asked whether and why students liked the challenge based lessons, 
how they were impacted, what difficulties they experienced, and for feedback for improve- 
ment. Students’ main ideas were summarized for each focus group and examined across 


for similarities and differences. 


Observation notes were running notes focused mainly on teachers’ instructional moves 
and overall lesson flow. These were used in conjunction with teacher interviews to sketch 
out the structures of the different challenge based lessons, which in turn were used to 
understand what the lessons looked like and where they can be improved. 


Lessons and student work were shared by the teachers during the site visits or electroni- 
cally. The lessons collected consisted mainly of student-facing material, such as handouts, 
slides, and readings. The student work collected were copies (or photos) of products that 
students had generated through the lesson, e.g., filled-out worksheets, projects, sketches 
in their notebooks. Supplementary information about the context, aim and flow of the 
lessons, was obtained through classroom observation (for all challenge based lessons, and 
some comparison lessons) and additional conversations with the teachers. 


As mentioned in the beginning of Section 3, four science teachers and two education 
researchers who were unfamiliar with the project scored each lesson and student work 

on the deeper learning and science content rubrics. Two of the teachers had more than 

10 years of experience teaching science in middle and upper elementary school, while the 
other two teachers were newer to their science teaching career (less than three years). All 
teachers had some experience teaching in Title | schools. Each scoring packet was a digital 
folder comprising of: a brief lesson overview created by researchers to look consistent 
across all lessons (including lesson topic, assignment description, student choice, duration, 
connection to prior lessons, description of group work involved if any, and grading criteria); 
lesson materials; and two student work samples. While most teachers submitted at least 
six samples of student work—two that exceeded their expectations, two that met their 
expectations, and two that did not meet their expectations—the scorers only analyzed 
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two among those that either met or exceeded expectations. We decided not to score 
lessons that “did not meet” expectations because (1) our main objective was to understand 
the learning opportunities and outcomes when students complete the assignment to a 
reasonable extent, and (2) we knew from our observations that the work that “didn’t meet 
teachers’ expectations” were generally exceptions, rather than the norm, in terms of the 
quality of work that students produced. 


The scorers spent two days at the Digital Promise office in San Mateo, CA, receiving training 
and scoring the lessons. All except one spent up to four additional hours scoring lessons 
remotely. The scoring order was randomized, and the same scorer who scored the lessons 
scored the student work. More than two-thirds of the lesson packets were scored by at 
least two scorers, and all except one lesson and more than 90 percent of the student work 
were scored by at least two scorers. 


The research team analyzed the scores and gathered additional feedback on the rubric 
dimensions (from scorers and additional middle school science experts) to assess whether 
the rubric scores were of adequate quality to use for this analysis. This validation analysis, 
documented in Iwatani, Vang, Romero, & Means (forthcoming), showed that the scorers, 
as a group, scored the lessons very consistently. The one-way random-effects average 
measures intraclass correlation (ICC) ranging from .87 to .94 for the deeper learning activity 
dimensions (all in the “excellent” range according to Cicchetti, 1994) and from .72 to .94 
for the deeper learning student work (either the “good” or “excellent” range). However, the 
ICC for the science content dimensions (Appendix D) were only .57 (fair) and .53 (fair) for 
activity and student work. The rubric dimension scores correlated with one another and 
with the science content scores in patterns that we expected based on theory and obser- 
vations, providing us with more confidence that the dimensions scores measured what 
we intended them to. Scorer comments and reviewers of the rubrics provided feedback 
on how specific elements can or should be improved, but on-the-whole agreed that the 
dimensions were important and clear, further indicating the trustworthiness of the scores 
for use in this research context. 


The rubric scores were averaged across scorers so that every lesson and student work had 
one score per dimension. Ratings of “N/A” were converted to “O" for this analysis since the 
population of interest for this project included all lessons (vs e.g., only secondary research 
lessons or only lessons that involved creating a product). We calculated mean scores for 
the three lesson types (typical, NGSS practices-aligned, and challenge based), and com- 
pared the means using one-way between-subjects analysis of variance (ANOVAs). To find 
the pattern of differences in rubric scores between the lesson types, we performed post 
hoc pairwise comparisons using the Bonferroni adjustment. For the two dimensions that 
violated the homogeneity of variance assumption (namely, the activity and student work 
ratings on critical secondary research), we performed Welch's test on the comparison to 
account for the violation. In addition, we calculated the effect size, Hedges’ g—a measure 
of how large the average score differences are in terms of standard deviation units—for 
each comparison. 
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It should be noted that initially we considered the challenge based lessons created in the 
first and second rounds of the project separately ("“CBL1” and “CBL2,” respectively). This 

was because we were not sure whether the two were different. Our initial ANOVA results 
and follow-up analyses indicated that the two groups of lessons were very similar in their 
characteristics, hence we considered the two groups as one (i.e., we compared the average 
of CBL1 and CBL2 with each of the comparison lessons) in subsequent analyses. 


Finally, to determine if there were differences in deeper learning opportunities between 
Title | and non-Title | schools, we conducted two-sample t-tests comparing the deeper 
learning opportunity scores between Title | and non-Title | schools for typical, NGSS 
practices-aligned, and challenge based science lessons. 


Data Tables 


In this section, we present the data tables pertaining to the findings mentioned in this 
report. For interpretations of the tables as well as additional analyses we conducted, see 
Iwatani, Vang, Romero, & Means (forthcoming). 


Tables Al presents descriptive statistics of each lesson type (i.e., typical, NGSS practic- 
es-aligned, and challenge based) for each rubric dimension. Tables A2 and A5 present one- 
way between-subjects ANOVA results for the deeper learning opportunities and student 
outcomes, respectively. Tables A3a and A3b present comparisons of deeper learning 
opportunities between challenge based and typical lessons, and challenge based and NGSS 
practices-aligned lessons, respectively. 


Table A4 presents descriptive statistics of each lesson type for the student outcomes (as 
seen in the student work). Table A5 presents one-way between-subjects ANOVA results 
for the student outcomes. Tables A6a and A6b show comparisons of student outcomes 
between challenge based and typical lessons, and challenge based and NGSS practic- 
es-aligned lessons, respectively. 


Table A7 displays the number of science lessons by lesson type for Title | and non-Title 

| schools. Table A8 presents the means and standard deviations for Title | and non-Title 

| schools by science lesson types. Results of comparison rubric scores between Title | 
and non-Title | schools for typical, NGSS practices-aligned, and challenge based science 
lessons are presented in Tables A9—A11. 
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Table A1. Activity rubric score means and standard deviations and by lesson type 


Learning opportunity Typical lessons Practices-aligned Challenge 
based 


A1. Real-world 
engagement 


A3. Practicing science & 
engineering 


A4. Critical secondary 
research 


A5. Substantive 
collaboration 


A6. Effective 
communication 


A1. Real-world engagement 


A3. Practicing science & 
engineering 


A4. Critical secondary eters ea 
research? 
eceees eee. [a [va i ae |e 
AG. Effective communication Effective communication “AG. Effectivecommunication | 3 | 67 | 508 | 00 | a9 a 
eee i i 


2 Welch's ANOVA used due to heterogeneity of variance, although n values were estimated from standard ANOVA. 
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Table A3a. Comparisons of deeper learning opportunities between “typical” and challenge based lessons 


Learning opportunity SE df Adjusted ES 
Pp? 
Ai. Real-world 143 0.25 67 5.82 < .001 ee 1.75 
engagement 
ro Co 


A3. Practicing science & 
engineering 


A4. Critical secondary 
research® 


A5. Substantive 
collaboration 


A6. Effective 
communication 


2 p-value doubled to control for Type | error. 
® Average of Hedges’ g for CBL 1 and CBL2. 
© Welch's ANOVA used due to heterogeneity of variance. 


Table A3b. Comparisons of deeper learning opportunities between “NGSS practices-aligned” and challenge based lessons 


A1. Real-world 
engagement 


A3. Practicing science & 
engineering 


AAé. Critical secondary 
research‘ 


A5. Substantive 
collaboration 


A6. Effective 
communication 


® p-value doubled to control for Type | error. 
® Average of Hedges’ g for CBL 1 and CBL2. 
© Welch's ANOVA used due to heterogeneity of variance. 
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Table A4. Average rubric scores and standard deviations for typical, practices-aligned, and 
challenge based lessons by student outcomes 


Student outcomes (as seen Typical lessons Practices- Challenge based 
in student work)t e aligned lessons lessons 


motenetayer see foe fife fame [ot 


S4. Conducted critical 
secondary research 
comnts ae aca to 


Understood disciplinary core 
ideas 


Table A5. One-way between-subjects ANOVA results on whether mean student work rubric 
scores differ across lesson type 


Student outcomes (as seen 
in student work) 


$1. Did something relevant 
S2. Practiced science 
$3. Used engineering design 


$4. Conducted critical 
secondary research? 


$5. Communicated effectively 


Understood disciplinary core 
ideas 


2 Welch's ANOVA used due to heterogeneity of variance, although n2 values were estimated from standard ANOVA. 
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Table A6a. Simple comparisons of student outcomes between typical and challenge based lessons 


Student outcomes (as 
seen in student work) 


$1. Did something relevant 
S2. Practiced science 


$5. Communicated 
effectively 


Understood disciplinary 
core ideas 


? p-value doubled to control for Type | error. 
» Average of Hedges’ g for CBL 1 and CBL2. 
© Welch's ANOVA used due to heterogeneity of variance. 


ome ls |e 
$4. Conducted critical eee | 
secondary research‘ 

fall il 


pire 


Table A6b. Simple comparisons of student outcomes between NGSS practices-aligned and challenge based lessons 


$1. Did something 
relevant 


S2. Practiced science 


Student outcomes (as 
seen in student work) 


$5. Communicated 
effectively 


Understood disciplinary -0.31 
core ideas 


2 p-value doubled to control for Type | error. 


$4. Conducted critical 
secondary research‘ 


® Average of Hedges’ g for CBL 1 and CBL2. 


© Welch's ANOVA used due to heterogeneity of variance. 


ee 
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Table A7. Number of lessons (by lesson type) in Title | and non-Title | schools 


Typical lessons Practices- 
aligned 


Table A8. Deeper learning scores and standard deviations for Title | and non-Title | schools by lesson type 


Learning Typical Practices- Challenge 
opportunity lessons aligned based 


re 
engagement 

wrist [one [ome [om [osm [oe 
wesarcon | vini—| e |a | ame [so [aie 

wnvini [se | ome [vo [ase [ne | 
wma [ri [om [om [ane [uo [ar | 0 
science & 
ee 
mone [ri [ove [om [one [os [aw | a0 
secondary 
ear [ vas [om [om [os [on [one [om 
wsazemee [ies | om [om [sco [ae [aa | 
collaboration 

worins [on [ome [202 [or [me [a 
went, [Ti |e [om [one [one [om [im 
communication 

wnvini [om [a8 [os [ae [| 0 
tienen [ei a5 [oma [se | om | oak 

womins [a [ome [aoe [ow [20 | 08 
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Table A9. Whether typical lessons differ in activity subric scores between Title | and non-Title | schools 


‘eonmpormnmy tarp | eer 
sirasencnssomnm oe fas |S fOr 


‘smccnme em asf 


Table A10. Whether NGSS practices-aligned lessons differ in activity rubric scores between Title | and non-Title | schools 
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Table A11. Whether challenge based lessons differ in activity rubric scores between Title | and non-Title | schools 


ce 


Digital Promise Deepening Science Engagement with Challenge Based Learning: Research Report | 61 


Appendix B. Activity Rubric 

These were used for the scoring described in this report. The updated versions based 
on feedback from the scorers and reliability analyses are available in Digital Promise’s 
Challenge Based Science Learning Toolkit. 


Activity Dimension 1: Activity calls for students to engage with real-world phenomena* or 
problems that connect with their interests and values. 


Activity 

doesn’t involve 
real-world 
phenomena or 
problems (e.g,., 

it asks students 
to learn facts 
oratheory ina 
decontextualized 
way). 


Examples: 


Worksheet on 
Punnett squares. 


Demonstration of 
evaporation and 
condensation 

of water. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity doesn't 
call for students 
to invest much 
time, emotion, 
or thinking 
towards the 
phenomenon 
or problem. 


Examples: 


Reading or 
watching a video 
about the impact 
of deforestation 
and answering 
questions. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity calls 
for students 

to invest time 
and thinking 
towards the 
phenomenon or 
problem. 


Activity doesn’t 
attempt to 
engage students 
emotionally. 


Examples: 


Designing the 
tallest building 
possible. 


Investigating 
how finches 
have evolved 
over time. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity calls 

for students 

to invest time 
and thinking 
towards the 
phenomenon or 
problem. 


Activity 
attempts to 
engage students 
emotionally by 
connecting with 
their interests 
and values. 


Examples: 


Designing a 
more effective 
layout for the 
school cafeteria. 


Investigating 
the nutritional 
content in chips 
vs vegetables. 


Activity involves 
a real-world 
phenomenon 
and/or problem. 


Activity calls 
for students 
to invest time 
and thinking 
towards the 
phenomenon 
or problem. 


Activity attempts 
to engage 
students 
emotionally by 
connecting with 
their interests 
and values. 


Learning about 
this topic can 
greatly expand 
students’ 
awareness, and 
change the way 
they think, feel, 
or act. 


Examples: 


Designing a 
more effective 
home layout 
for a local 
resident who is 
paraplegic. 


Investigating 
the nutritional 
content in food 
that students 
want to learn 
more about. 


*Note: “Phenomena are observable events in nature (or our lives) that connect to multiple NGSS disciplinary core ideas, such as Finnish Snow 


Trees or the behavior of bees” (Maltese, n.d.). 
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Activity Dimension 2: Activity calls for students to guide their own learning of science or engineering. 


Activity calls 
for students 
to follow the 
teacher's lead 
every step of 
the way. 


Examples: 


Students read and 
answer questions 
about the reading. 


Activity calls 

for students 

to make 
surface-level 
decisions about 
their learning 
experience. 


The decisions 
aren't directly 
tied to science 
or engineering. 


Examples: 


Students are 
prompted to 
choose color, 
font, style, 
layout, or 
order of simple 
sub-tasks. 


Students are told 
they may listen 
to music if the 
want, and do 
their work in the 
library 

or outside. 


Activity calls 

for students 

to make a few 
short-term 
decisions about 
their learning 
experience. 


The decisions 
are directly tied 
to science or 
engineering. 


These decisions 
don't really 
impact what, 
how, and to 
what degree 
students will 
learn the 
science or 
engineering. 


Examples: 


Students can 
choose which 
species they cite 
as an illustrative 
example of a 
predator, but 
they don't do 
much research 
on them. 


Students can 
choose among 
different 
available 
materials, 
product types, 
or optimization 
processes that 
yield similar 
outcomes. 


Activity calls 
for students 

to make a 
substantive 
decision that 
would impact 
what, how or 
to what degree 
they learn the 
science, or how 
they engineer 
solutions. 


Students are 
not prompted 
to reflect 

on and take 
responsibility 
for their own 
learning (i.e., 
not provided the 
opportunity to 
reflect on and 
adjust 

their decisions). 


Examples: 


Students can 
choose a 
science learning 
topic, question, 
goal, path, tool, 
or product. 


Students 

can choose 
engineering 
problem, 
SUCCESS 
criteria, design 
constraints, and 
development 
and/or 
optimization 
process, 

which make a 
difference to the 


solution's quality. 


Activity calls for 
students to make 
a substantive 
decision about 
what, how, or 

to what degree 
they learn the 
science, or how 
they engineer 
solutions. 


They are 
prompted 

to take 
responsibility 
for their own 
learning (i.e., 
to reflect on 
and adjust 
their decisions, 
making course 
corrections 

as necessary). 


Examples: 


Students are 
prompted to do 
what's described 
on the column 
to the left, and 
to brainstorm 
pros and cons 
of their decision 
through check- 
ins with teacher 
or classmates. 


Students 

are asked to 
produce a 
summative 
reflection on 
decisions they 
made and 
their impact. 
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Activity Dimension 3: Activity calls for students to discover principles of effective designs through direct experience. 


Activity does 

not call for 
students to learn 
about scientific 
principles or 
engineering 
designs. 


OR 


Activity calls for 
students to just 
read, hear, or 
watch videos 
about scientific 
principles or 
engineering 
designs, without 
experiencing 
them directly. 


Examples: 


Practicing skills 
they already 
know. 


Lecture, reading, 
and/or video on 
soil erosion. 


Activity calls for 
students to get 
some first-hand 
experience 
related to 

the principle 

/ design, but 

it is either in 
the form of 
demonstration 


or confirmation. 


Examples: 


Students 
learn about 
conservation 
of mass, and 
do a lab that 
illustrates it. 


Activity calls 

for students 

to discover 
underlying 
principles / 
effective designs 
through direct 
experience. 


Activity is 
excessively 
scaffolded (e.g., 
all students are 
guaranteed 

the discovery 

if they follow 
directions). 


Examples: 


Lab or design 
task with 
step-by-step 
instructions 
so discovery is 
guaranteed. 


“Aimless design” 


Activity calls 
for students 
to discover 
underlying 
principles / 
effective designs 
through direct 
experience, 
without 
excessive 
scaffolding. 


It doesn’t call 
for students 
to articulate 
connections 
between the 
activity and 
scientific 
principles, 

or test the 
effectiveness of 
their designs. 


Examples: 


Lab where 
students actually 
play/explore, 
and have to 
express what 
they learned. 


Activity calls 
for students 
to discover 
underlying 
principles / 
effective designs 
through direct 
experience, 
without 
excessive 
scaffolding. 


It calls for 
students to 
articulate the 
connection 
between the 
principles 

and their 
experience, or 
the implications 
of their test 
results to 
their design. 


Examples: 


What's on the 
left column, 
but with added 
requirement 
to discuss 
limitations to 
discovery/ 
design. 


64 | Deepening Science Engagement with Challenge Based Learning: Research Report 


Digital Promise 


Activity Dimension 4: Activity calls for students to conduct critical background research on their own. 


Activity does 
not ask students 
to obtain 
information 


OR 


Activity calls 

for students 

to obtain 
information from 
just one source or 
does not specify 
that information 
must come from 
multiple sources. 


OR 


Assignment is not 
complex enough 
that multiple 
sources are not 
needed. 


Activity calls 
for students 
to obtain 
information 
from multiple 
sources. 


Activity calls 
for students 
to obtain 
information 
from multiple 
appropriate 
sources. 


It asks students 
to combine 
information 
from multiple 
sources to 

form answers 
expressed using 
his/her own 
words. 


Activity calls 
for students 
to obtain 
information 
from multiple 
appropriate 
sources. 


It asks students 
to combine the 
information from 
multiple sources 
to explain a 
phenomenon 

or suggest a 
solution toa 
design problem. 


Activity calls 
for students 
to obtain 
information 
from multiple 
appropriate 
SOUICES. 


It asks students 
to combine the 
information from 
multiple sources 
to explain a 
phenomenon 

or suggest a 
solution toa 
design problem. 


It asks students 
to attend to 

the credibility, 
accuracy, and 
possible bias of 
each publication 
and method 
used. 
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Activity Dimension 5: Activity calls for students to collaborate substantively. 


Activity does not 
call for students 
to work together. 


Activity calls 

for students to 
work together, 
but does not call 
for students to 
create a shared 
product(s). 


Activity calls 

for students to 
work together 
towards a shared 
product(s). 

For many 
groups, the 
product(s) can 
be completed 
with one round 
of discussion 
and decision- 
making (e.g., on 
what to do and 
who does 
which part). 


Activity calls for 
students to work 
towards a shared 
product(s). 

The product 

is sufficiently 
complex such 
that for most 
student groups, 
one round of 
discussion is 
likely to be 
insufficient for 
its successful 
completion. 


Activity calls 
for students to 
work towards a 
shared product. 
The product 

is sufficiently 
complex that 
multiple rounds 
of agreement 
on what-to-do 
and who-does- 
what-part are 
necessary for 
the successful 
completion of 
the product. 


In addition, 
activity 
specifically calls 
for all students 
to explicitly 
attend their 
process and/ 

or efficacy of 
collaboration 
(e.g., asking 
them to 
establish group 
norms or reflect 
on the extent 

to which they 
collaborated 
successfully). 
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Activity Dimension 6: Activity calls for students to effectively organize, style, and format their communication. 


Activity does 

not involve 
communicating 
information to an 
audience. 


OR 


Activity calls 
for students to 
communicate 
simple 
information. 


Examples: 


Educational 
games; Taking 
notes 


Multiple choice 
and short answer 
questions. 


Activity calls 
for students to 
communicate 
simple 
information. 


It calls for 
students to 
effectively 
format and style 
the information. 


Examples: 


Poster 
conveying 
the message 
that smoking 
is harmful for 
your lungs, 
that follows 
certain format 
guidelines. 


Slideshow 
illustrating key 
vocabulary 
terms of a unit 
that's assessed 
on format. 


Activity calls 
for students to 
communicate 
fairly complex 
or nuanced 
information. 


It calls for 
students to 
effectively 
format and style 
the information. 


Examples: 


Draw and 
explain the 
process of 
mitosis. Drawing 
and explanation 
must follow 
certain format 
guidelines (e.g., 
clear, legible, 
color-coded). 


Activity calls 
for students to 
communicate 
fairly complex 
or nuanced 
information. 


It calls for 
students to 
effectively 
format, style, 
and organize 
(structure) their 


communication. 


Examples: 


Essay on how 
pollution affects 
meiosis of fish 
(where students 
must combine 
and organize 
the topics 
themselves). 


Activity calls 
for students to 
communicate 
a substantial 
amount of 
information. 


It calls for 
students to 
effectively 
format, style, 
and organize 
(structure) their 
communication. 


It calls for 
students to 
improve their 
communication 
approach based 
on feedback. 


Examples: 


Debate or 
moderated panel 
discussion ona 
scientific topic. 


Lessons that 
involve feedback 
sessions (by 
teacher or 
student) as 

part of the 
developmental 
process for 
communicating 
information. 


Note: This is a generic rubric that can be used to assess a variety of communication products (e.g., presentations, videos, and posters). 
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Appendix C. Student Work Rubric 


These were used for the scoring described in this report. The updated versions based 


on feedback from the scorers and reliability analyses are available in Digital Promise’s 


Challenge Based Science Learning Toolkit. 


Student Work Dimension 1: Student(s) did/created something potentially valuable to others as part of their learning. 


Student 
product is 
missing, 
incomplete, 
and/or 
misses 

the point. 


Assignment 
didn't 

ask for 
student(s) 
to create 
something. 


Examples: Examples: 


Notes, 
guided labs. 


A poster 
about 
weather, 
when the 
assignment 
was about 
genetics. 


A product 
was Created, 
but its 
interest/ 
value to 
others is 
questionable 
(e.g., purpose 
is unclear 


or not 
compelling). 


Examples: 


A poster that 
shows the 
stages of 
mitosis. 


A word 
search that 
contains 
genetics 
terms. 


Student did/ 
created 
something that 
could be of 
interest/ value 
to others. 


But the 
product is not 
particularly 
original or 
creative. 

It mainly 
repurposes 
already 
available 
information in 
a simple way. 


Examples: 


Educational 
game or poster 
that raises 
awareness 
about 
genetically 
modified 
foods. 


Student did/ 
created 
something that 
could be of 
interest/value 
to others. 


The student's 
work is 
original or 
creative. 


Examples: 


A prototype 
of an app 
that supports 
parents with 
a genetic 
mutation 
making 
decisions 
about having 
a child. 


Student did/ 
created 
something 
that could be 
of interest/ 
value to 
others. 


The student's 
work is 
original or 
creative. 


In addition, 
student 
provided 
evidence that 
either (1) they 
thoughtfully 
reflected on 
strengths and 
weaknesses 
of their 
design and/ 
or (2) their 
product was 
appreciated 
by others. 


Examples: 


The app 
(mentioned to 
the left), with 
a letter of 
support from 
a user, and/ 
ora written 
reflection on 
strengths and 
weaknesses of 
the product. 
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Student Work Dimension 2: Student(s) learned through practicing science 


Nota 
science 
lesson. 


Student 
didn't do 
anything that 
resembles 

a scientific 
investigation 
(.e., an 
empirical/ 
theoretical 
experiment, 
dissection, or 
fieldwork). 


Examples: 


Notes on 
genes 


Practice 
problems on 
balancing 
equations 


Student did a 
small aspect 
of a scientific 
investigation. 


Examples: 


Extended 
analysis 
questions on 
genetics. 


Using 
simulations 
that help 
conceptualize 
molecular 
motion. 


Close 
observation 
of a squirrel's 
behavior for 
five minutes 
without 
making 
meaning. 


Student did 
a majority of 
a scientific 
investigation. 


Examples: 


Students 
record data 
and come 
up with 
inferences. 


Students 
dissect a squid. 


Close 
observation 
of a squirrel’s 
behavior and 
come up 
with some 
inferences 
or scientific 
questions. 


Student did 
a majority of 
a scientific 
investigation. 


In addition, 
student made 
reference 

to a larger 
(scientific 

or practical) 
question to 
which their 
investigation 
was relevant. 


Examples: 


Used analysis 
of molecular 
structure of 
chemicals 

in pesticides 
to help 
recommend 
the best 
pesticide for 
the school 
garden. 


Relied on 
experimental 
relationship 
between 
temperature 


and pressure to 


argue how it’s 
important to 
attend to the 
thickness of 

a space 
shuttle’s wall. 


Student did 
a majority of 
a scientific 
investigation. 


In addition, 
student made 
reference 

to a larger 
(scientific 

or practical) 
question to 
which their 
investigation 
was relevant. 


In addition, 
student 
clearly and 
correctly 
explained 
how their 
scientific 
investigation 
informed 
their larger 
question. 


Example: 


Student 
does what's 
described 
under #3, 
and clearly 
conveys how 
the solution 
to their bigger 
question is 
informed by 
their science 
investigation. 


Note: NGSS practices are described here. For this dimension, exclude “obtaining and communicating information,” which is assessed in S4. 
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Student Work Dimension 3: Student(s) made sound use of the three major stages of the 
engineering design process (i.e., define, develop solutions, optimize) 


engineering 
lesson 


Student 
didn’t use any 
part of the 
engineering 
design 
process. 


Example: 


Proposed a 
solution to 

a problem 
without any 
research 
about 

the problem. 


Student 
successfully 
conducted 
some aspects 
of the 
engineering 
design process 
but did not 
create an 


actual product. 


Example: 


Generated 
ideas and/or 
researched 
what might 
be relevant to 
designing 

a solution. 


Student 
successfully 
conducted 
some aspects 
of the 
engineering 


design process. 


Student 
created a 
prototype 
or product. 


Student 
successfully 
conducted 
some 

aspects of an 
engineering 
design process. 


Student 
created a 
prototype or 
product. 


The prototype 
or product is 
accompanied 
with some 
descriptions/ 
specifications 
of the design. 


Student 
successfully 
conducted 
most aspects 
of an 
engineering 
design 
process. 


Student 
created a 
prototype or 
product. 


The prototype 
or product is 
accompanied 
by 
compelling 
descriptions/ 
specifications 
of the design 
(e.g., reasons 
the design 

is good, 
explanation 
that the 
product is 
based ona 
systematic 
design 
process). 
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Student Work Dimension 4. Student(s) critically researched existing information 


(Aligned with NGSS middle school standards on obtaining information: https://ngss.nsta.org/Practices.aspx?id=8) 


Student was 
not asked 
to research 
existing 
information. 


Student 
obtained 
information 
from only a 
single source. 


Student 
obtained 
information 
from multiple 
sources. 


Some sources 
were not 
appropriate 
or student 
simply copied 
information. 


Student 
obtained 
information 
from multiple 
sources. 


Most/all 
sources were 
appropriate. 


Student 
combined 
information 
from multiple 
sources to 
form answers 
expressed 
using his/her 
own words. 


Student 
obtained 
information 
from multiple 
sources. 


All sources 
were 
appropriate. 


Student 
combined the 
information 
from multiple 
sources to 
explain a 
phenomenon 
or suggest 

a solution 

to a design 
problem. 


Student obtained 
information from 
multiple sources. 


All sources were 
appropriate. 


Student 
combined the 
information from 
multiple sources 
to explain a 
phenomenon 

or suggest a 
solution to a 
design problem. 


Student 
demonstrated 
attention to 

the credibility, 
accuracy, and 
possible bias of 
the publications 
and methods 
used. 


Student Work Dimension 5: The organization, style, and format of the student's work is effective. 


Student(s) 
were not 
tasked with 
work where 


organization, 
style, and 
format 


matter. 


The work is 
incomplete, 
and/or so 
disorganized 
that the 
teacher/ 
audience 
cannot tell 
what i 

going on. 


The work is 
complete. 
There is an 
attempt at 
organization, 


but it’s not very 


effective. 


Organization 
of the work 
makes sense, 


but the format 
and style is not 
very effective. 


Effective 
organization, 
format and 
style. 


Meets 
expectations. 


Very effective 
organization, 
format and style. 


Exceeds 
expectations. 


Note: This is a generic rubric that can be used to assess a variety of communication products, including presentations, video products, artwork and 
posters. For classroom use, teachers should provide students in advance with additional details on what constitutes “meets expectations,” based on 


the assignment. 
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Appendix D. Rubrics Used to Assess Opportunities and 
Outcomes of Learning Grade-Appropriate Disciplinary 
Core Ideas 


Activity Dimension on Science Content: Activity calls for students to understand grade-appropriate 
disciplinary core ideas. 


This dimension is about alignment of content Knowledge to the NGSS disciplinary core 
ideas. To use this dimension, it’s necessary to assess how the content maps to relevant 
NGSS disciplinary core ideas from the list here. 


Activity is 
unrelated to any 
of the disciplinary 
core ideas that 
middle school 
students are 
expected to learn. 


Example: 


Activity on how 
to use an app to 
create cartoons. 


Activity is related 
to disciplinary 
core ideas, but 
it’s unclear what 
they are. 


Example: 


Egg drop 
experiment with 
no discussion on 
what the point 
of it was. 


Activity calls 

for students 

to learn 
disciplinary core 
ideas, but the 
scope, level, or 
scaffolds are off 
(e.g., too many/ 
few standards; 
expectations are 
too high or 

too low). 


Example: 


Middle school 
students needing 
to calculate 

of carrying 
capacity. 


A lesson that 
takes a week 

to convey only 
that gravitational 
forces (only) 
attract. 


Activity calls 

for students to 
learn disciplinary 
core ideas at 

an appropriate 
pace, scope, 
and inclusion of 
scaffolds. 


Activity calls 

for students to 
learn disciplinary 
core ideas at an 
appropriate pace 
and scope. 


Activity is 
impressive in 
its inclusion of 
core ideas. 


Example: 


Activity that 
effectively, 
efficiently, 
or creatively 
includes 
core ideas. 
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Student Work Dimension on Science Content: Student(s) demonstrated a sound understanding of relevant disciplinary 
core ideas 


To use this rubric, it’s necessary to separately articulate the level of understanding that is 
expected at the middle school level, for example, by selecting the relevant NGSS disci- 
plinary core ideas from the list here. 


Students were 
not asked to 
demonstrate 
the 
connection 
of their work 
with relevant 
content 
knowledge. 


Demonstrates 
little/no 
understanding 
of the relevant 
disciplinary 
core ideas. 


Serious errors. 


Demonstrates 
some 
understanding 
of the relevant 
disciplinary 
core ideas. 


At least one 
major error. 


Demonstrates 
some 
understanding 
of the relevant 
disciplinary core 
ideas. 


No major 
errors. 


Demonstrates 
a solid 
understanding 
of the relevant 
disciplinary 
core ideas. 


No major 
errors. 


Demonstrates 
a solid 
understanding 
of the relevant 
disciplinary 
core ideas. 


No major 
errors. 


Understanding 
is deeper or 
more nuanced 
than would 

be expected 

at the middle 
school level. 
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Appendix E. List of OER Found and Explored by Project Teachers 


OER found Brief description What teachers liked/disliked 


Phenomena Database Database of NGSS science Vast quantity, but still easy to 
phenomena explore 


Greenhouse Effect Phet simulation on greenhouse Easy to understand. Good visual 
Simulator effect learning piece 


https://phet.colorado.edu/ Phet simulation on color vision Easy to use. Additional resource 
sims/html/color-vision/ to support how to interpret light 
latest/color-vision_en.html and color. 


https://www.learner.org/ Teacher resources and PD across Variety of plans/topics 
the curriculum by Annenberg 
Learner (division of Annenberg 
Foundation) 


https://www. Science related videos, GIFs, Used in the past, variety of bio 
amoebasisters.com/ comics, etc., created by the related videos and handouts to 
Amoeba Sisters, who “love accompany videos 
empowering teacher and student 
creators” and want to demystify 
science. 


Stanford NGSS Integrated “Integrated, NGSS-designed, Middle school lessons. Difficult to 
Curriculum by SCALE project-based curriculum for 7th identify whether or not it's open 
and 8th grade." 


https://www.oercommons. gr6-9 lesson for bacteria in food Middle school lesson for bacteria 
org/courses/who-s- in food 
hitchhiking-in-your-food 


Animal Hearing (PBS & OER) Shark senses compared to other 
animals’ senses 
Animal hearing 
Video on hearing loss (humans) 


Open Professionals A resource hub with links to many A central place to use as a search 

Education Network resources that are OER platform for different websites 
that provide OER videos, 
pictures, websites, etc. 


OER Commons Big repository of OER Good search features 


Open4us.org TED talks, clipart, and other Gives you what is approved by 
resources that is open OER 


Filters on YouTube/Google to Filter tends to retrieve results of 
get “Creative Commons" uneven quality 
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